2024-03-25T16:20:26.926 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:20:26.926 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/orig.config.yaml 2024-03-25T16:20:26.969 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:20:27.053 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621272' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8BW/raJ7hs8HkgBcrqosAu5QNhsG8VKh8OtuWAIjVOj/xngIyLwFiFziQPbI2DGtRp/gvAIt4u1l82IIosaco= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9c2Q9gPXMZsTiZJDGLRFyhiZQQwT0ovlYf6cy2hhSfX+o3b6NEreqMXCwN6ywjkx7gSe/R9PjyQUF8U/yPY4o= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:20:27.054 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:20:27.054 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:20:27.055 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:20:27.056 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:20:27.080 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:20:27.080 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:20:27.080 INFO:teuthology.packaging:ref: None 2024-03-25T16:20:27.080 INFO:teuthology.packaging:tag: None 2024-03-25T16:20:27.080 INFO:teuthology.packaging:branch: main 2024-03-25T16:20:27.080 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:20:27.080 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:20:27.406 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:20:27.408 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:20:27.415 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:20:27.415 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:20:27.429 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:20:27.446 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:20:27.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:20:27.477 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:15:33.770599', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8BW/raJ7hs8HkgBcrqosAu5QNhsG8VKh8OtuWAIjVOj/xngIyLwFiFziQPbI2DGtRp/gvAIt4u1l82IIosaco='} 2024-03-25T16:20:27.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:15:33.769316', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9c2Q9gPXMZsTiZJDGLRFyhiZQQwT0ovlYf6cy2hhSfX+o3b6NEreqMXCwN6ywjkx7gSe/R9PjyQUF8U/yPY4o='} 2024-03-25T16:20:27.499 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:20:27.505 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:20:27.505 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:20:27.505 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:20:27.581 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd9f0a670d0>, signals=[15]) 2024-03-25T16:20:27.581 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:20:27.588 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:20:27.588 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T16:20:27.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:20:27.697 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T16:20:27.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:20:27.786 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:20:27.793 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-03-25T16:20:27.810 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-03-25T16:20:27.810 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-03-25T16:20:27.866 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="CentOS Stream" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="9" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:ID="centos" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="9" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:ANSI_COLOR="0;31" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:20:27.867 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:20:27.868 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:20:27.868 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:20:27.868 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-03-25T16:20:27.894 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-03-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-03-25T16:20:27.912 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:20:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:20:27.970 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:20:27.970 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:20:27.970 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:20:27.970 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-03-25T16:20:27.994 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:20:28.003 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:20:28.008 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:20:28.009 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:20:28.011 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:20:28.027 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:20:28.033 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:20:28.033 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:20:28.066 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:20:28.092 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:20:28.148 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:20:28.164 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:20:28.164 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:20:28.164 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:20:28.165 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:20:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:20:28.180 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-03-25T16:20:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:20:28.181 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-03-25T16:20:28.237 INFO:teuthology.orchestra.run.smithi042.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:20:28.237 INFO:teuthology.task.kernel:Running kernel on smithi042: 5.14.0-361.el9.x86_64 2024-03-25T16:20:28.237 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum install -y kernel 2024-03-25T16:20:28.238 INFO:teuthology.orchestra.run.smithi120.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:20:28.239 INFO:teuthology.task.kernel:Running kernel on smithi120: 5.14.0-361.el9.x86_64 2024-03-25T16:20:28.239 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum install -y kernel 2024-03-25T16:20:30.311 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:20:30.423 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:20:31.670 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-25T16:20:31.890 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-25T16:20:32.611 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:20:33.039 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-25T16:20:34.558 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-25T16:20:36.157 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:02 2024-03-25T16:20:38.590 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-25T16:20:40.108 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-25T16:20:41.113 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-03-25T16:20:42.600 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T16:20:42.609 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T16:20:44.554 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:20:47.590 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-25T16:20:49.710 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T16:20:49.730 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:20:49.730 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:20:49.875 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:20:49.886 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:20:49.886 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-03-25T16:20:49.887 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:20:49.887 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:20:49.887 INFO:teuthology.orchestra.run.smithi120.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:20:49.887 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:20:49.887 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:20:49.887 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout:Install 4 Packages 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 76 M 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 123 M 2024-03-25T16:20:49.888 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:20:50.141 INFO:teuthology.orchestra.run.smithi120.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 513 kB/s | 44 kB 00:00 2024-03-25T16:20:50.975 INFO:teuthology.orchestra.run.smithi120.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-25T16:20:51.858 INFO:teuthology.orchestra.run.smithi120.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 19 MB/s | 33 MB 00:01 2024-03-25T16:20:51.934 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:20:51.934 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:20:52.027 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:20:52.036 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout: Package Architecture Version Repository Size 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout:Install 4 Packages 2024-03-25T16:20:52.037 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:20:52.038 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 76 M 2024-03-25T16:20:52.038 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 123 M 2024-03-25T16:20:52.038 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:20:52.417 INFO:teuthology.orchestra.run.smithi120.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T16:20:52.417 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:52.417 INFO:teuthology.orchestra.run.smithi120.stdout:Total 30 MB/s | 76 MB 00:02 2024-03-25T16:20:52.437 INFO:teuthology.orchestra.run.smithi042.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 353 kB/s | 44 kB 00:00 2024-03-25T16:20:53.054 INFO:teuthology.orchestra.run.smithi042.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-25T16:20:53.078 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:20:53.466 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:20:53.467 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:20:53.712 INFO:teuthology.orchestra.run.smithi042.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 24 MB/s | 33 MB 00:01 2024-03-25T16:20:54.146 INFO:teuthology.orchestra.run.smithi042.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-25T16:20:54.146 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:54.146 INFO:teuthology.orchestra.run.smithi042.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-25T16:20:54.470 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:20:54.470 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:20:54.838 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:20:55.257 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:20:55.257 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:20:56.298 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:20:56.298 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:20:56.947 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:20:57.549 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:20:57.644 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:20:58.894 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:20:59.534 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:20:59.630 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:21:00.404 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:21:00.543 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:21:02.508 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:21:02.644 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:21:05.273 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:21:05.368 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:07.720 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:21:07.761 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:10.130 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:12.854 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:39.532 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:39.532 INFO:teuthology.orchestra.run.smithi120.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:21:39.533 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:21:39.576 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:21:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:21:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:21:40.339 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:40.339 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:21:40.339 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:21:40.339 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:21:40.339 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:21:40.339 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:21:40.339 INFO:teuthology.orchestra.run.smithi120.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:21:40.340 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:21:40.340 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:21:40.701 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:21:40.798 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:21:40.798 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:21:40.798 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:21:40.799 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:21:40.799 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:21:40.799 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:21:40.799 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi120.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:21:40.799 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:21:40.877 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:21:40.878 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:21:40.878 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -qi grub2-tools 2024-03-25T16:21:40.918 INFO:teuthology.orchestra.run.smithi120.stdout:Name : grub2-tools 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Epoch : 1 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Version : 2.06 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Release : 68.el9 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: x86_64 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Group : Unspecified 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Size : 8264254 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:License : GPLv3+ 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Packager : builder@centos.org 2024-03-25T16:21:40.919 INFO:teuthology.orchestra.run.smithi120.stdout:Vendor : CentOS 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:Summary : Support tools for GRUB. 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:Description : 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:hardware devices. 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:21:40.920 INFO:teuthology.orchestra.run.smithi120.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:21:40.921 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:21:40.921 INFO:teuthology.task.kernel:Updating grub on smithi120 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:21:40.921 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:21:41.547 INFO:teuthology.orchestra.run.smithi120.stderr:Generating grub configuration file ... 2024-03-25T16:21:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:21:42.914 INFO:teuthology.orchestra.run.smithi120.stderr:done 2024-03-25T16:21:42.919 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-25T16:21:42.934 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.yNpUjK4lUF 2024-03-25T16:21:42.935 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yNpUjK4lUF 2024-03-25T16:21:43.003 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.yNpUjK4lUF 2024-03-25T16:21:43.143 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.yNpUjK4lUF is 6KB 2024-03-25T16:21:43.191 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.yNpUjK4lUF 2024-03-25T16:21:43.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:21:43.282 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:21:43.282 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:21:43.282 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:21:43.282 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:21:43.283 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:21:43.436 DEBUG:teuthology.orchestra.run.smithi120:> sudo shutdown -r now 2024-03-25T16:21:43.610 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:43.610 INFO:teuthology.orchestra.run.smithi042.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:21:43.610 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:21:43.656 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:44.059 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:44.059 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:21:44.060 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:21:44.060 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:21:44.484 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:21:44.485 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:21:44.814 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:21:44.920 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:21:44.921 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:21:44.921 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:21:44.921 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:21:44.922 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:21:44.922 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:21:44.922 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:21:44.922 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:21:45.005 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:21:45.005 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:21:45.006 DEBUG:teuthology.orchestra.run.smithi042:> sudo rpm -qi grub2-tools 2024-03-25T16:21:45.043 INFO:teuthology.orchestra.run.smithi042.stdout:Name : grub2-tools 2024-03-25T16:21:45.043 INFO:teuthology.orchestra.run.smithi042.stdout:Epoch : 1 2024-03-25T16:21:45.043 INFO:teuthology.orchestra.run.smithi042.stdout:Version : 2.06 2024-03-25T16:21:45.043 INFO:teuthology.orchestra.run.smithi042.stdout:Release : 68.el9 2024-03-25T16:21:45.043 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: x86_64 2024-03-25T16:21:45.043 INFO:teuthology.orchestra.run.smithi042.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:21:45.043 INFO:teuthology.orchestra.run.smithi042.stdout:Group : Unspecified 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Size : 8264254 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:License : GPLv3+ 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Packager : builder@centos.org 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Vendor : CentOS 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Summary : Support tools for GRUB. 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:Description : 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:21:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:21:45.045 INFO:teuthology.orchestra.run.smithi042.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:21:45.045 INFO:teuthology.orchestra.run.smithi042.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:21:45.045 INFO:teuthology.orchestra.run.smithi042.stdout:hardware devices. 2024-03-25T16:21:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:21:45.045 INFO:teuthology.orchestra.run.smithi042.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:21:45.046 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:21:45.046 INFO:teuthology.task.kernel:Updating grub on smithi042 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:21:45.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:21:45.658 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2024-03-25T16:21:47.024 INFO:teuthology.orchestra.run.smithi042.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:21:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:done 2024-03-25T16:21:47.067 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-03-25T16:21:47.083 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.RivlsHKMmD 2024-03-25T16:21:47.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RivlsHKMmD 2024-03-25T16:21:47.156 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.RivlsHKMmD 2024-03-25T16:21:47.298 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.RivlsHKMmD is 6KB 2024-03-25T16:21:47.347 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.RivlsHKMmD 2024-03-25T16:21:47.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:21:47.430 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:21:47.430 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:21:47.430 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:21:47.431 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:21:47.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:21:47.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2024-03-25T16:22:13.462 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:22:13.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-25T16:22:13.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:17.590 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:22:17.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-25T16:22:17.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:31.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2024-03-25T16:22:36.066 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.42 2024-03-25T16:22:40.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-25T16:22:40.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:44.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.120 2024-03-25T16:22:45.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-25T16:22:45.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:56.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-25T16:22:56.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:27.856 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-25T16:23:28.234 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-25T16:23:28.235 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:23:28.235 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-03-25T16:23:28.250 INFO:teuthology.orchestra.run.smithi120.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:23:28.250 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:23:28.250 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:23:28.250 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:23:29.251 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:23:29.251 DEBUG:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:23:29.337 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2024-03-25T16:23:29.383 DEBUG:teuthology.parallel:result is None 2024-03-25T16:23:45.071 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:23:57.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-25T16:23:57.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:57.367 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-03-25T16:23:57.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-25T16:23:57.759 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:23:57.759 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-03-25T16:23:57.774 INFO:teuthology.orchestra.run.smithi042.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:23:57.774 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:23:57.774 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:23:57.774 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:23:58.775 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:23:58.775 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:23:58.841 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2024-03-25T16:23:58.873 DEBUG:teuthology.parallel:result is None 2024-03-25T16:23:58.873 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:23:58.883 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:23:58.883 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:23:58.885 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:23:58.903 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:23:58.911 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:23:58.921 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:23:58.921 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:23:58.941 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:23:58.969 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:23:58.977 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:23:58.977 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:23:59.006 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:23:59.040 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:23:59.047 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:23:59.054 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:23:59.060 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:23:59.062 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:23:59.070 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:23:59.070 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:23:59.098 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:23:59.132 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:23:59.141 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:23:59.141 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:23:59.167 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:23:59.188 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:23:59.265 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:23:59.312 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:23:59.392 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:23:59.436 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T16:23:59.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:23:59.504 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:23:59.549 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:23:59.597 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:23:59.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:23:59.724 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T16:23:59.724 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:23:59.793 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-03-25T16:23:59.796 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-03-25T16:23:59.844 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:23:59.889 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:24:00.340 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:24:00.348 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:24:00.348 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:24:00.360 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:24:00.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo service auditd rotate 2024-03-25T16:24:00.424 INFO:teuthology.orchestra.run.smithi042.stdout:Rotating logs: 2024-03-25T16:24:00.426 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-03-25T16:24:00.491 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-03-25T16:24:00.493 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:24:00.493 DEBUG:teuthology.orchestra.run.smithi042:> /usr/sbin/getenforce 2024-03-25T16:24:00.522 INFO:teuthology.orchestra.run.smithi042.stdout:Permissive 2024-03-25T16:24:00.522 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-03-25T16:24:00.547 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-03-25T16:24:00.547 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi042.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:24:00.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:24:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:00.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:24:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:00.618 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:24:00.618 DEBUG:teuthology.orchestra.run.smithi042:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:24:00.664 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:24:00.698 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:24:00.707 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:24:00.726 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:24:00.727 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:28:21.801 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2024-03-25T16:28:21.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-25T16:28:21.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:21.897 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-03-25T16:28:21.981 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-25T16:28:21.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-25T16:28:21.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:22.069 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-25T16:28:22.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-25T16:28:22.146 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:28:22.157 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:28:22.157 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:28:22.157 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:28:22.160 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:28:22.160 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:28:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:28:22.205 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:28:22.217 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:28:22.234 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:28:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: ntpd: command not found 2024-03-25T16:28:22.249 INFO:teuthology.orchestra.run.smithi042.stdout:506 Cannot talk to daemon 2024-03-25T16:28:22.263 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-03-25T16:28:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:28:22.276 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-03-25T16:28:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:28:22.293 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:28:22.309 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:28:22.334 INFO:teuthology.orchestra.run.smithi042.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:28:22.338 INFO:teuthology.orchestra.run.smithi042.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:28:22.338 INFO:teuthology.orchestra.run.smithi042.stdout:=============================================================================== 2024-03-25T16:28:22.338 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.338 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.338 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.338 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.359 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:28:22.362 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:28:22.362 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-25T16:28:22.362 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.362 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.362 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.363 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:28:22.363 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:28:22.371 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:28:22.371 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:28:22.371 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:28:22.372 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:28:22.378 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:28:22.378 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:28:22.378 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:28:22.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:28:22.381 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:28:22.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:28:22.522 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:28:22.522 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:28:22.531 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:28:22.531 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:28:22.964 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:28:22.964 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T16:28:22.964 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:28:22.984 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:28:22.984 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T16:28:22.984 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:28:22.996 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:28:22.996 DEBUG:teuthology.orchestra.run.smithi042:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:28:23.013 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:28:23.014 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:28:23.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:28:23.084 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:28:23.147 DEBUG:teuthology.orchestra.run.smithi042:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:28:23.166 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:28:23.214 INFO:teuthology.orchestra.run.smithi042.stdout:check_obsoletes = 1 2024-03-25T16:28:23.216 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum clean all 2024-03-25T16:28:23.233 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-03-25T16:28:23.234 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-03-25T16:28:23.462 INFO:teuthology.orchestra.run.smithi042.stdout:74 files removed 2024-03-25T16:28:23.469 INFO:teuthology.orchestra.run.smithi120.stdout:74 files removed 2024-03-25T16:28:23.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-radosgw 2024-03-25T16:28:23.495 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw 2024-03-25T16:28:24.777 INFO:teuthology.orchestra.run.smithi042.stdout:ceph packages for x86_64 83 kB/s | 82 kB 00:00 2024-03-25T16:28:24.789 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 82 kB/s | 82 kB 00:00 2024-03-25T16:28:25.686 INFO:teuthology.orchestra.run.smithi042.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:28:26.156 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 7.6 kB/s | 10 kB 00:01 2024-03-25T16:28:26.612 INFO:teuthology.orchestra.run.smithi042.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T16:28:27.090 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-25T16:28:28.025 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 7.5 MB/s | 6.7 MB 00:00 2024-03-25T16:28:28.306 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - AppStream 4.0 MB/s | 6.7 MB 00:01 2024-03-25T16:28:29.873 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-25T16:28:29.909 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-03-25T16:28:32.296 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-25T16:28:33.207 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-25T16:28:34.081 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-25T16:28:35.175 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-25T16:28:39.257 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-25T16:28:40.426 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.1 MB 00:01 2024-03-25T16:28:41.274 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 16 kB 00:00 2024-03-25T16:28:42.093 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-25T16:28:42.334 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T16:28:43.863 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T16:28:47.435 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-03-25T16:28:48.819 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T16:28:49.983 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:28:49.994 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:28:49.994 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:28:49.994 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:28:49.994 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:28:49.994 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:28:49.994 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrading: 2024-03-25T16:28:49.994 INFO:teuthology.orchestra.run.smithi042.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: boost-program-options 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:28:49.995 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:28:49.996 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout:Installing weak dependencies: 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:28:49.997 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:28:49.998 INFO:teuthology.orchestra.run.smithi042.stdout:Install 30 Packages 2024-03-25T16:28:49.998 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrade 2 Packages 2024-03-25T16:28:49.998 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:28:49.998 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 58 M 2024-03-25T16:28:49.998 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:28:51.298 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options 2024-03-25T16:28:51.318 INFO:teuthology.orchestra.run.smithi120.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:28:51.319 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:28:51.320 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout:Install 30 Packages 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 2 Packages 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 58 M 2024-03-25T16:28:51.321 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:29:02.378 INFO:teuthology.orchestra.run.smithi120.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 505 kB/s | 5.3 MB 00:10 2024-03-25T16:29:02.469 INFO:teuthology.orchestra.run.smithi120.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:29:03.994 INFO:teuthology.orchestra.run.smithi120.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 486 kB/s | 742 kB 00:01 2024-03-25T16:29:05.020 INFO:teuthology.orchestra.run.smithi120.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 486 kB/s | 498 kB 00:01 2024-03-25T16:29:05.026 INFO:teuthology.orchestra.run.smithi042.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 371 kB/s | 5.3 MB 00:14 2024-03-25T16:29:05.160 INFO:teuthology.orchestra.run.smithi042.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 189 kB/s | 25 kB 00:00 2024-03-25T16:29:07.910 INFO:teuthology.orchestra.run.smithi042.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 270 kB/s | 742 kB 00:02 2024-03-25T16:29:09.668 INFO:teuthology.orchestra.run.smithi042.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 283 kB/s | 498 kB 00:01 2024-03-25T16:29:20.169 INFO:teuthology.orchestra.run.smithi120.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 345 kB/s | 5.1 MB 00:15 2024-03-25T16:29:20.269 INFO:teuthology.orchestra.run.smithi120.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T16:29:20.602 INFO:teuthology.orchestra.run.smithi120.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 394 kB/s | 131 kB 00:00 2024-03-25T16:29:21.002 INFO:teuthology.orchestra.run.smithi120.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 405 kB/s | 162 kB 00:00 2024-03-25T16:29:21.711 INFO:teuthology.orchestra.run.smithi120.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 452 kB/s | 320 kB 00:00 2024-03-25T16:29:22.253 INFO:teuthology.orchestra.run.smithi120.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 550 kB/s | 298 kB 00:00 2024-03-25T16:29:22.470 INFO:teuthology.orchestra.run.smithi120.stdout:(11/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 322 kB/s | 9.7 MB 00:30 2024-03-25T16:29:22.537 INFO:teuthology.orchestra.run.smithi120.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 350 kB/s | 99 kB 00:00 2024-03-25T16:29:22.620 INFO:teuthology.orchestra.run.smithi120.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 708 kB/s | 106 kB 00:00 2024-03-25T16:29:22.645 INFO:teuthology.orchestra.run.smithi120.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-25T16:29:22.671 INFO:teuthology.orchestra.run.smithi120.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-25T16:29:22.721 INFO:teuthology.orchestra.run.smithi120.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T16:29:22.746 INFO:teuthology.orchestra.run.smithi120.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T16:29:22.771 INFO:teuthology.orchestra.run.smithi120.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 832 kB/s | 41 kB 00:00 2024-03-25T16:29:22.863 INFO:teuthology.orchestra.run.smithi120.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 381 kB/s | 35 kB 00:00 2024-03-25T16:29:22.922 INFO:teuthology.orchestra.run.smithi120.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.2 MB/s | 216 kB 00:00 2024-03-25T16:29:22.947 INFO:teuthology.orchestra.run.smithi120.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 532 kB/s | 44 kB 00:00 2024-03-25T16:29:23.131 INFO:teuthology.orchestra.run.smithi120.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-03-25T16:29:23.182 INFO:teuthology.orchestra.run.smithi120.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-03-25T16:29:23.225 INFO:teuthology.orchestra.run.smithi120.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-03-25T16:29:23.257 INFO:teuthology.orchestra.run.smithi120.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-25T16:29:23.366 INFO:teuthology.orchestra.run.smithi120.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-25T16:29:23.391 INFO:teuthology.orchestra.run.smithi120.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 838 kB 00:00 2024-03-25T16:29:23.416 INFO:teuthology.orchestra.run.smithi120.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-25T16:29:23.500 INFO:teuthology.orchestra.run.smithi120.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-25T16:29:27.293 INFO:teuthology.orchestra.run.smithi042.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 269 kB/s | 9.7 MB 00:37 2024-03-25T16:29:27.651 INFO:teuthology.orchestra.run.smithi042.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 126 kB/s | 45 kB 00:00 2024-03-25T16:29:28.235 INFO:teuthology.orchestra.run.smithi042.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 225 kB/s | 131 kB 00:00 2024-03-25T16:29:28.808 INFO:teuthology.orchestra.run.smithi120.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 641 kB/s | 3.4 MB 00:05 2024-03-25T16:29:28.893 INFO:teuthology.orchestra.run.smithi042.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 272 kB/s | 5.1 MB 00:19 2024-03-25T16:29:28.918 INFO:teuthology.orchestra.run.smithi042.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 237 kB/s | 162 kB 00:00 2024-03-25T16:29:29.485 INFO:teuthology.orchestra.run.smithi042.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 541 kB/s | 320 kB 00:00 2024-03-25T16:29:29.902 INFO:teuthology.orchestra.run.smithi042.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 303 kB/s | 298 kB 00:00 2024-03-25T16:29:30.003 INFO:teuthology.orchestra.run.smithi042.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 192 kB/s | 99 kB 00:00 2024-03-25T16:29:30.054 INFO:teuthology.orchestra.run.smithi042.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 707 kB/s | 106 kB 00:00 2024-03-25T16:29:30.088 INFO:teuthology.orchestra.run.smithi042.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-25T16:29:30.107 INFO:teuthology.orchestra.run.smithi042.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-25T16:29:30.154 INFO:teuthology.orchestra.run.smithi042.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T16:29:30.179 INFO:teuthology.orchestra.run.smithi042.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T16:29:30.221 INFO:teuthology.orchestra.run.smithi042.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 620 kB/s | 41 kB 00:00 2024-03-25T16:29:30.271 INFO:teuthology.orchestra.run.smithi042.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-03-25T16:29:30.313 INFO:teuthology.orchestra.run.smithi042.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.6 MB/s | 216 kB 00:00 2024-03-25T16:29:30.338 INFO:teuthology.orchestra.run.smithi042.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 665 kB/s | 44 kB 00:00 2024-03-25T16:29:30.705 INFO:teuthology.orchestra.run.smithi042.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 788 kB/s | 309 kB 00:00 2024-03-25T16:29:30.780 INFO:teuthology.orchestra.run.smithi042.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-25T16:29:30.864 INFO:teuthology.orchestra.run.smithi042.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-03-25T16:29:30.948 INFO:teuthology.orchestra.run.smithi042.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 810 kB/s | 67 kB 00:00 2024-03-25T16:29:31.098 INFO:teuthology.orchestra.run.smithi042.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-25T16:29:31.182 INFO:teuthology.orchestra.run.smithi042.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-25T16:29:31.349 INFO:teuthology.orchestra.run.smithi042.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-25T16:29:32.016 INFO:teuthology.orchestra.run.smithi042.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 4.4 MB 00:01 2024-03-25T16:29:33.724 INFO:teuthology.orchestra.run.smithi120.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 313 kB/s | 3.1 MB 00:10 2024-03-25T16:29:33.924 INFO:teuthology.orchestra.run.smithi042.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 466 kB/s | 20 MB 00:43 2024-03-25T16:29:37.232 INFO:teuthology.orchestra.run.smithi120.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 446 kB/s | 20 MB 00:45 2024-03-25T16:29:37.237 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:37.237 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.3 MB/s | 58 MB 00:45 2024-03-25T16:29:37.269 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:29:37.307 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:29:37.307 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:29:37.778 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:29:37.778 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:29:38.766 INFO:teuthology.orchestra.run.smithi042.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 475 kB/s | 3.1 MB 00:06 2024-03-25T16:29:39.256 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:29:39.339 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:29:39.370 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:29:39.402 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:29:39.435 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:39.450 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:39.698 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:39.756 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:40.089 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:29:40.155 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:40.248 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:40.336 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:29:40.373 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:40.427 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:40.464 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:29:40.508 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:29:40.577 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:29:40.600 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:40.646 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:40.673 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:29:40.783 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:29:40.825 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:29:40.865 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:29:40.898 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:29:40.927 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:29:40.960 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:29:41.269 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:29:41.297 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:41.371 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:41.402 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:29:41.444 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:29:41.782 INFO:teuthology.orchestra.run.smithi042.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 331 kB/s | 3.4 MB 00:10 2024-03-25T16:29:41.786 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:41.786 INFO:teuthology.orchestra.run.smithi042.stdout:Total 1.1 MB/s | 58 MB 00:51 2024-03-25T16:29:41.818 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:29:41.860 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:29:41.860 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:29:41.894 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:29:41.925 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:41.979 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:42.047 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:29:42.077 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:29:42.350 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:29:42.351 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:29:43.967 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:29:44.051 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:29:44.078 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:29:44.104 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:29:44.135 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:44.151 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:44.233 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:44.273 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:44.293 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:44.350 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:44.698 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:29:44.747 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:44.814 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:44.831 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:44.868 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:44.931 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:29:44.946 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:44.946 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:29:44.946 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:29:44.946 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:29:44.965 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:44.986 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:45.028 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:29:45.102 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:29:45.170 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:29:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:45.255 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:45.283 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:29:45.393 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:29:45.434 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:29:45.474 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:29:45.507 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:29:45.536 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:29:45.560 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:29:45.889 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:29:45.923 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:46.014 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:46.045 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:29:46.087 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:29:46.569 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:29:46.592 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:46.647 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:29:46.753 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:29:48.998 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:49.042 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:49.610 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:49.645 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:49.739 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:49.760 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:29:49.760 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:29:49.760 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:29:49.779 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:54.463 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:54.463 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-03-25T16:29:54.464 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-03-25T16:29:54.464 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-03-25T16:29:54.464 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-03-25T16:29:54.464 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-03-25T16:29:54.464 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-03-25T16:29:54.464 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-03-25T16:29:54.464 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:29:55.402 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:29:55.425 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:29:55.425 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:55.425 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:29:55.425 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:29:55.425 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:29:55.426 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:29:55.427 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:29:55.512 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:29:55.512 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:56.616 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:29:56.617 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:29:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:29:56.619 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:56.619 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:29:56.619 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:29:57.516 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:57.516 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:29:57.517 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.518 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:29:57.519 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:29:57.819 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-test 2024-03-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:01:10 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:29:58.520 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:29:58.521 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:29:58.522 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:29:58.522 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:29:58.522 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:29:58.522 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:29:58.522 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2024-03-25T16:29:58.522 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:29:58.523 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 48 M 2024-03-25T16:29:58.523 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 204 M 2024-03-25T16:29:58.523 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:29:58.692 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 217 kB 00:00 2024-03-25T16:29:58.801 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.7 MB/s | 300 kB 00:00 2024-03-25T16:29:58.835 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:29:58.968 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:29:59.510 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 173 kB/s | 163 kB 00:00 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /sys 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /proc 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /mnt 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /var/tmp 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /home 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /root 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /tmp 2024-03-25T16:29:59.548 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:30:00.485 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:30:00.523 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:30:00.524 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:30:00.524 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:30:00.524 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:30:00.524 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:30:00.524 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:30:00.526 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:30:00.613 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:30:00.613 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:30:01.655 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:30:01.655 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:30:01.655 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:30:01.655 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:30:01.656 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:30:01.656 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:30:01.656 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:30:01.656 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:30:01.656 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:30:01.657 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:30:01.657 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:30:01.657 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:30:01.658 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:30:01.658 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:30:01.658 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:30:01.658 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:30:01.659 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:30:01.659 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:30:01.659 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:30:01.659 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:30:01.659 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:30:01.660 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:30:01.660 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:30:01.660 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:30:01.660 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:30:01.660 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:30:01.661 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:30:01.661 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:30:01.661 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:30:01.661 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:30:01.661 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:30:01.661 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:30:01.662 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:30:01.662 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:30:02.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:30:02.793 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:30:02.793 INFO:teuthology.orchestra.run.smithi042.stdout:Upgraded: 2024-03-25T16:30:02.793 INFO:teuthology.orchestra.run.smithi042.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.793 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.793 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:30:02.794 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:02.795 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:30:02.796 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:30:02.796 INFO:teuthology.orchestra.run.smithi042.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:30:02.796 INFO:teuthology.orchestra.run.smithi042.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:30:02.796 INFO:teuthology.orchestra.run.smithi042.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:30:02.796 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:30:02.796 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:30:03.133 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-test 2024-03-25T16:30:03.713 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:30:03.867 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:30:03.867 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:30:03.868 INFO:teuthology.orchestra.run.smithi042.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:30:03.869 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:30:03.869 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:30:03.869 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:30:03.869 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:30:03.869 INFO:teuthology.orchestra.run.smithi042.stdout:Install 6 Packages 2024-03-25T16:30:03.869 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:30:03.870 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 48 M 2024-03-25T16:30:03.870 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 204 M 2024-03-25T16:30:03.870 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:30:04.143 INFO:teuthology.orchestra.run.smithi042.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T16:30:04.220 INFO:teuthology.orchestra.run.smithi042.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.5 MB/s | 300 kB 00:00 2024-03-25T16:30:04.254 INFO:teuthology.orchestra.run.smithi042.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:30:04.396 INFO:teuthology.orchestra.run.smithi042.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T16:30:04.821 INFO:teuthology.orchestra.run.smithi042.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 199 kB/s | 163 kB 00:00 2024-03-25T16:32:35.457 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 309 kB/s | 47 MB 02:36 2024-03-25T16:32:35.458 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:35.459 INFO:teuthology.orchestra.run.smithi120.stdout:Total 314 kB/s | 48 MB 02:36 2024-03-25T16:32:35.470 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:32:35.491 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:32:35.491 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:32:35.876 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:32:35.876 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:32:36.156 INFO:teuthology.orchestra.run.smithi042.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 318 kB/s | 47 MB 02:32 2024-03-25T16:32:36.158 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:36.159 INFO:teuthology.orchestra.run.smithi042.stdout:Total 324 kB/s | 48 MB 02:32 2024-03-25T16:32:36.169 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:32:36.191 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:32:36.462 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:32:36.534 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:32:36.573 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:32:36.588 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:32:36.588 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:32:36.613 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:32:36.656 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:32:36.684 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:32:37.197 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:32:37.274 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:32:37.325 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:32:37.365 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:32:37.450 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:32:41.134 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:32:41.325 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:32:42.168 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:32:42.244 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:32:43.745 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:32:43.745 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:32:43.745 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:32:43.745 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:32:43.746 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:32:43.746 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:32:44.109 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:32:44.110 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:32:44.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph 2024-03-25T16:32:44.552 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:32:44.552 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:32:44.552 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:32:44.552 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:32:44.552 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:32:44.552 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:32:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:32:44.924 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:32:44.958 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:03:56 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:32:45.117 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:32:45.119 INFO:teuthology.orchestra.run.smithi120.stdout:========================================================================================= 2024-03-25T16:32:45.119 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:32:45.119 INFO:teuthology.orchestra.run.smithi120.stdout:========================================================================================= 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:32:45.120 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:32:45.121 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:32:45.122 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:32:45.123 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:32:45.123 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:32:45.123 INFO:teuthology.orchestra.run.smithi120.stdout:========================================================================================= 2024-03-25T16:32:45.123 INFO:teuthology.orchestra.run.smithi120.stdout:Install 36 Packages 2024-03-25T16:32:45.123 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:32:45.123 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 30 M 2024-03-25T16:32:45.124 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 107 M 2024-03-25T16:32:45.124 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:32:45.231 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph 2024-03-25T16:32:45.800 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:03:58 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:32:45.819 INFO:teuthology.orchestra.run.smithi120.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:32:45.961 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout:========================================================================================= 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout:========================================================================================= 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:32:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:32:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:32:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout:========================================================================================= 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout:Install 36 Packages 2024-03-25T16:32:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:32:45.968 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 30 M 2024-03-25T16:32:45.968 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 107 M 2024-03-25T16:32:45.968 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:32:46.800 INFO:teuthology.orchestra.run.smithi042.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T16:32:50.851 INFO:teuthology.orchestra.run.smithi042.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 343 kB/s | 1.5 MB 00:04 2024-03-25T16:32:51.653 INFO:teuthology.orchestra.run.smithi120.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 243 kB/s | 1.5 MB 00:06 2024-03-25T16:32:54.384 INFO:teuthology.orchestra.run.smithi042.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 272 kB/s | 2.1 MB 00:07 2024-03-25T16:32:55.051 INFO:teuthology.orchestra.run.smithi042.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 371 kB/s | 247 kB 00:00 2024-03-25T16:32:55.176 INFO:teuthology.orchestra.run.smithi042.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:32:55.218 INFO:teuthology.orchestra.run.smithi042.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:32:55.251 INFO:teuthology.orchestra.run.smithi042.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-25T16:32:55.285 INFO:teuthology.orchestra.run.smithi042.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:32:55.318 INFO:teuthology.orchestra.run.smithi042.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:32:55.360 INFO:teuthology.orchestra.run.smithi042.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T16:32:55.436 INFO:teuthology.orchestra.run.smithi042.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:32:55.469 INFO:teuthology.orchestra.run.smithi042.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:32:55.503 INFO:teuthology.orchestra.run.smithi042.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:32:55.537 INFO:teuthology.orchestra.run.smithi042.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:32:55.679 INFO:teuthology.orchestra.run.smithi042.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 882 kB/s | 125 kB 00:00 2024-03-25T16:32:55.771 INFO:teuthology.orchestra.run.smithi042.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-25T16:32:56.129 INFO:teuthology.orchestra.run.smithi042.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 422 kB/s | 151 kB 00:00 2024-03-25T16:32:56.221 INFO:teuthology.orchestra.run.smithi042.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-25T16:32:56.321 INFO:teuthology.orchestra.run.smithi042.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T16:32:56.455 INFO:teuthology.orchestra.run.smithi042.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-25T16:32:56.547 INFO:teuthology.orchestra.run.smithi042.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-03-25T16:32:56.622 INFO:teuthology.orchestra.run.smithi042.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T16:32:56.722 INFO:teuthology.orchestra.run.smithi042.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 232 kB/s | 23 kB 00:00 2024-03-25T16:32:56.798 INFO:teuthology.orchestra.run.smithi042.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-03-25T16:32:56.827 INFO:teuthology.orchestra.run.smithi120.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 190 kB/s | 2.1 MB 00:11 2024-03-25T16:32:56.898 INFO:teuthology.orchestra.run.smithi042.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 197 kB/s | 20 kB 00:00 2024-03-25T16:32:56.991 INFO:teuthology.orchestra.run.smithi042.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 505 kB/s | 46 kB 00:00 2024-03-25T16:32:57.065 INFO:teuthology.orchestra.run.smithi042.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T16:32:57.150 INFO:teuthology.orchestra.run.smithi042.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-25T16:32:57.226 INFO:teuthology.orchestra.run.smithi042.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2024-03-25T16:32:57.291 INFO:teuthology.orchestra.run.smithi042.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-25T16:32:57.367 INFO:teuthology.orchestra.run.smithi042.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-03-25T16:32:57.450 INFO:teuthology.orchestra.run.smithi042.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T16:32:57.559 INFO:teuthology.orchestra.run.smithi042.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-03-25T16:32:57.626 INFO:teuthology.orchestra.run.smithi042.stdout:(34/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 443 kB/s | 4.7 MB 00:10 2024-03-25T16:32:57.642 INFO:teuthology.orchestra.run.smithi042.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-25T16:32:58.179 INFO:teuthology.orchestra.run.smithi120.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 183 kB/s | 247 kB 00:01 2024-03-25T16:32:58.294 INFO:teuthology.orchestra.run.smithi120.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:32:58.336 INFO:teuthology.orchestra.run.smithi120.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:32:58.370 INFO:teuthology.orchestra.run.smithi120.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-25T16:32:58.404 INFO:teuthology.orchestra.run.smithi120.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:32:58.437 INFO:teuthology.orchestra.run.smithi120.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:32:58.479 INFO:teuthology.orchestra.run.smithi120.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:32:58.554 INFO:teuthology.orchestra.run.smithi120.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:32:58.588 INFO:teuthology.orchestra.run.smithi120.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:32:58.622 INFO:teuthology.orchestra.run.smithi120.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:32:58.664 INFO:teuthology.orchestra.run.smithi120.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:32:58.789 INFO:teuthology.orchestra.run.smithi120.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T16:32:58.931 INFO:teuthology.orchestra.run.smithi120.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-25T16:32:59.482 INFO:teuthology.orchestra.run.smithi120.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 274 kB/s | 151 kB 00:00 2024-03-25T16:32:59.599 INFO:teuthology.orchestra.run.smithi120.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 373 kB/s | 43 kB 00:00 2024-03-25T16:32:59.766 INFO:teuthology.orchestra.run.smithi120.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-25T16:32:59.966 INFO:teuthology.orchestra.run.smithi120.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T16:33:00.066 INFO:teuthology.orchestra.run.smithi120.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T16:33:00.158 INFO:teuthology.orchestra.run.smithi120.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T16:33:00.259 INFO:teuthology.orchestra.run.smithi120.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-25T16:33:00.359 INFO:teuthology.orchestra.run.smithi120.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T16:33:00.459 INFO:teuthology.orchestra.run.smithi120.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 197 kB/s | 20 kB 00:00 2024-03-25T16:33:00.559 INFO:teuthology.orchestra.run.smithi120.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T16:33:00.668 INFO:teuthology.orchestra.run.smithi120.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 728 kB/s | 79 kB 00:00 2024-03-25T16:33:00.810 INFO:teuthology.orchestra.run.smithi120.stdout:(28/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 320 kB/s | 4.7 MB 00:14 2024-03-25T16:33:00.835 INFO:teuthology.orchestra.run.smithi120.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-25T16:33:00.936 INFO:teuthology.orchestra.run.smithi120.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 897 kB/s | 90 kB 00:00 2024-03-25T16:33:01.036 INFO:teuthology.orchestra.run.smithi120.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-25T16:33:01.153 INFO:teuthology.orchestra.run.smithi120.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-25T16:33:01.178 INFO:teuthology.orchestra.run.smithi120.stdout:(33/36): python3-portend-3.1.0-2.el9.noarch.rpm 45 kB/s | 16 kB 00:00 2024-03-25T16:33:01.304 INFO:teuthology.orchestra.run.smithi120.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-03-25T16:33:01.329 INFO:teuthology.orchestra.run.smithi120.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 133 kB/s | 20 kB 00:00 2024-03-25T16:33:29.990 INFO:teuthology.orchestra.run.smithi042.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 439 kB/s | 17 MB 00:39 2024-03-25T16:33:29.994 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:29.994 INFO:teuthology.orchestra.run.smithi042.stdout:Total 690 kB/s | 30 MB 00:44 2024-03-25T16:33:30.016 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:33:30.061 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:33:30.061 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:33:30.472 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:33:30.473 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:33:31.813 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:33:31.867 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:33:31.909 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:33:31.936 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:33:32.011 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:33:32.061 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:33:32.093 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:33:32.190 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:33:32.267 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:33:32.313 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:33:32.388 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:33:32.440 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:33:32.477 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:33:32.538 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:33:32.600 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:33:32.797 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:33:32.848 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:33:32.886 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:33:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:33:32.951 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:33:33.031 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:33:33.069 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:33:33.103 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:33:33.144 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:33:33.178 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:33:33.213 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:33:33.308 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:33:33.344 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:33:33.419 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:33:33.564 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:33:33.693 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:33:33.859 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:33:33.890 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:33:33.915 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:33:33.915 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:33.915 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:33:33.915 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:33:33.915 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:33:33.915 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:35.424 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:33:35.450 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:33:35.450 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:35.450 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:33:35.450 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:33:35.450 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:33:35.451 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:35.916 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:33:35.939 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:33:35.939 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:35.939 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:33:35.939 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:33:35.939 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:33:35.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:36.159 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:33:36.181 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:33:36.181 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:36.181 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:33:36.181 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:33:36.182 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:33:36.182 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:36.267 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:33:37.975 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:33:37.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:33:37.977 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:33:37.977 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:33:37.977 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:33:37.977 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:33:37.977 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:33:37.977 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:33:37.978 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:33:37.978 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:33:37.978 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:33:37.978 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:33:37.978 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:33:37.978 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:33:38.877 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:33:38.877 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:33:38.878 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:33:38.879 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:38.880 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:33:39.259 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-base 2024-03-25T16:33:39.837 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:33:39.938 INFO:teuthology.orchestra.run.smithi042.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:39.992 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:33:39.996 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:33:39.996 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:33:40.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install cephadm 2024-03-25T16:33:40.635 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:33:40.790 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:33:40.791 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:33:40.791 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:33:40.791 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:33:40.791 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:33:40.791 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:33:40.791 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:40.791 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:33:40.792 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:33:40.792 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:33:40.792 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:40.792 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 747 k 2024-03-25T16:33:40.792 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 754 k 2024-03-25T16:33:40.792 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:33:44.208 INFO:teuthology.orchestra.run.smithi042.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 219 kB/s | 747 kB 00:03 2024-03-25T16:33:44.208 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Total 219 kB/s | 747 kB 00:03 2024-03-25T16:33:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:33:44.216 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:33:44.216 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:33:44.222 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:33:44.222 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:33:44.299 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:33:44.862 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:44.949 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:45.296 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:45.586 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:45.587 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:45.587 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:33:45.587 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:45.587 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:45.587 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:33:45.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:33:46.553 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 145 k 2024-03-25T16:33:46.554 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 436 k 2024-03-25T16:33:46.555 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:33:47.431 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 166 kB/s | 145 kB 00:00 2024-03-25T16:33:47.431 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:47.432 INFO:teuthology.orchestra.run.smithi042.stdout:Total 166 kB/s | 145 kB 00:00 2024-03-25T16:33:47.432 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:33:47.441 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:33:47.441 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:33:47.499 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:33:47.499 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:33:47.648 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:33:47.715 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:33:47.739 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:33:47.739 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:47.739 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:33:47.739 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:48.689 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:33:48.689 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:48.689 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:33:48.689 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:48.689 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:48.689 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:33:48.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr 2024-03-25T16:33:49.515 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:33:49.617 INFO:teuthology.orchestra.run.smithi042.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:49.672 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:33:49.673 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:33:49.673 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:33:49.742 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:33:50.313 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:33:50.469 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:33:50.470 INFO:teuthology.orchestra.run.smithi042.stdout:====================================================================================== 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout:====================================================================================== 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:33:50.471 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout:====================================================================================== 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout:Install 12 Packages 2024-03-25T16:33:50.472 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:50.473 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 6.9 M 2024-03-25T16:33:50.473 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 26 M 2024-03-25T16:33:50.473 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:33:51.626 INFO:teuthology.orchestra.run.smithi042.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 40 kB/s | 31 kB 00:00 2024-03-25T16:33:52.234 INFO:teuthology.orchestra.run.smithi042.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 11 kB/s | 15 kB 00:01 2024-03-25T16:33:52.426 INFO:teuthology.orchestra.run.smithi042.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-25T16:33:52.569 INFO:teuthology.orchestra.run.smithi042.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.0 MB/s | 863 kB 00:00 2024-03-25T16:33:52.860 INFO:teuthology.orchestra.run.smithi042.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:33:52.902 INFO:teuthology.orchestra.run.smithi042.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 809 kB/s | 1.0 MB 00:01 2024-03-25T16:33:52.927 INFO:teuthology.orchestra.run.smithi042.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 295 kB/s | 19 kB 00:00 2024-03-25T16:33:53.078 INFO:teuthology.orchestra.run.smithi042.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-25T16:33:53.103 INFO:teuthology.orchestra.run.smithi042.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 825 kB/s | 144 kB 00:00 2024-03-25T16:33:53.128 INFO:teuthology.orchestra.run.smithi042.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 618 kB/s | 31 kB 00:00 2024-03-25T16:33:53.162 INFO:teuthology.orchestra.run.smithi042.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-25T16:33:53.589 INFO:teuthology.orchestra.run.smithi120.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 277 kB/s | 17 MB 01:01 2024-03-25T16:33:53.595 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:53.595 INFO:teuthology.orchestra.run.smithi120.stdout:Total 444 kB/s | 30 MB 01:08 2024-03-25T16:33:53.617 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:33:53.662 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:33:53.662 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:33:54.090 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:33:54.091 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:33:55.414 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:33:55.482 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:33:55.527 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:33:55.552 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:33:55.622 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:33:55.659 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:33:55.693 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:33:55.784 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:33:55.887 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:33:55.903 INFO:teuthology.orchestra.run.smithi042.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 360 kB/s | 1.8 MB 00:05 2024-03-25T16:33:55.909 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:55.909 INFO:teuthology.orchestra.run.smithi042.stdout:Total 1.3 MB/s | 6.9 MB 00:05 2024-03-25T16:33:55.928 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:33:55.929 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:33:55.949 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:33:55.949 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:33:56.000 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:33:56.054 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:33:56.092 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:33:56.133 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:33:56.133 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:33:56.141 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:33:56.197 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:33:56.381 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:33:56.430 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:33:56.460 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:33:56.495 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:33:56.525 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:33:56.601 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:33:56.644 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:33:56.649 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:33:56.677 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:33:56.718 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:33:56.729 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:33:56.753 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:33:56.773 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:33:56.788 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:33:56.820 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:33:56.849 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:33:56.876 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:33:56.910 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:33:56.958 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:33:56.982 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:33:57.087 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:33:57.127 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:33:57.179 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:33:57.221 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:33:57.249 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:33:57.254 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:33:57.281 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:33:57.410 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:33:57.441 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:33:57.466 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:33:57.466 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:57.466 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:33:57.466 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:33:57.467 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:33:57.467 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:33:57.543 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:33:57.589 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:33:58.391 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:33:58.737 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:33:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:33:58.739 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:33:58.739 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:58.739 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:33:58.909 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:33:58.938 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:33:58.938 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:58.938 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:33:58.938 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:33:58.938 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:33:58.938 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:33:58.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:33:59.393 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:33:59.417 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:33:59.489 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:59.489 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:33:59.489 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:33:59.489 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:33:59.489 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:33:59.562 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:33:59.627 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:33:59.650 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:33:59.650 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:59.650 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:33:59.650 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:33:59.650 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:33:59.650 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:33:59.709 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:33:59.718 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================================ 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================================ 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:33:59.720 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:33:59.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:33:59.722 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:59.722 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:33:59.722 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================================ 2024-03-25T16:33:59.722 INFO:teuthology.orchestra.run.smithi042.stdout:Install 12 Packages 2024-03-25T16:33:59.722 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:33:59.722 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 37 M 2024-03-25T16:33:59.723 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 209 M 2024-03-25T16:33:59.723 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:00.160 INFO:teuthology.orchestra.run.smithi042.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 148 kB/s | 32 kB 00:00 2024-03-25T16:34:00.202 INFO:teuthology.orchestra.run.smithi042.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 413 kB/s | 17 kB 00:00 2024-03-25T16:34:00.268 INFO:teuthology.orchestra.run.smithi042.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 513 kB/s | 34 kB 00:00 2024-03-25T16:34:00.394 INFO:teuthology.orchestra.run.smithi042.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.7 MB/s | 3.0 MB 00:00 2024-03-25T16:34:01.445 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:34:01.445 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:34:01.445 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:34:01.445 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:34:01.446 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:34:01.447 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:34:01.447 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:34:01.447 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:34:01.447 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:34:01.447 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:34:01.448 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:34:01.449 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:34:01.449 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:34:01.449 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:34:01.449 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:34:01.919 INFO:teuthology.orchestra.run.smithi042.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 2.9 MB/s | 4.7 MB 00:01 2024-03-25T16:34:02.053 INFO:teuthology.orchestra.run.smithi042.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.9 MB/s | 4.8 MB 00:01 2024-03-25T16:34:02.078 INFO:teuthology.orchestra.run.smithi042.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2024-03-25T16:34:02.397 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:34:02.397 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:02.397 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:34:02.397 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:02.397 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:34:02.398 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:34:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:02.400 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:02.770 INFO:teuthology.orchestra.run.smithi042.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-25T16:34:02.825 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base 2024-03-25T16:34:02.920 INFO:teuthology.orchestra.run.smithi042.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T16:34:03.070 INFO:teuthology.orchestra.run.smithi042.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-25T16:34:03.287 INFO:teuthology.orchestra.run.smithi042.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-03-25T16:34:03.384 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:03.480 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:03.531 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:03.532 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:34:03.532 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:03.601 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install cephadm 2024-03-25T16:34:04.157 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:04.304 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:34:04.305 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:34:04.306 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:04.306 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 747 k 2024-03-25T16:34:04.306 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 754 k 2024-03-25T16:34:04.306 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:34:06.911 INFO:teuthology.orchestra.run.smithi120.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 287 kB/s | 747 kB 00:02 2024-03-25T16:34:06.912 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:06.912 INFO:teuthology.orchestra.run.smithi120.stdout:Total 287 kB/s | 747 kB 00:02 2024-03-25T16:34:06.913 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:34:06.919 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:34:06.919 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:34:06.925 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:34:06.925 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:34:07.011 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:34:07.440 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:34:07.525 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:34:07.886 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:34:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:34:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:08.179 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:34:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:08.180 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:08.367 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:34:08.938 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:09.088 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:34:09.089 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:34:09.090 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:34:09.090 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:09.090 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 145 k 2024-03-25T16:34:09.090 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 436 k 2024-03-25T16:34:09.090 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:34:10.077 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 147 kB/s | 145 kB 00:00 2024-03-25T16:34:10.078 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:10.079 INFO:teuthology.orchestra.run.smithi120.stdout:Total 147 kB/s | 145 kB 00:00 2024-03-25T16:34:10.079 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:34:10.088 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:34:10.088 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:34:10.140 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:34:10.141 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:34:10.281 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:34:10.345 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:34:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:34:10.369 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:10.369 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:34:10.369 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:11.362 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:34:11.362 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:11.362 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:34:11.362 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:11.362 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:11.362 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:11.615 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr 2024-03-25T16:34:12.186 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:12.286 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:12.338 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:12.339 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:34:12.339 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:12.409 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:34:12.570 INFO:teuthology.orchestra.run.smithi042.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 599 kB/s | 7.4 MB 00:12 2024-03-25T16:34:12.574 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:12.574 INFO:teuthology.orchestra.run.smithi042.stdout:Total 2.9 MB/s | 37 MB 00:12 2024-03-25T16:34:12.617 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:12.643 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:12.643 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:12.964 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:13.018 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:13.018 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:13.112 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:13.113 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-03-25T16:34:13.113 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:34:13.113 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-03-25T16:34:13.113 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:34:13.114 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:34:13.115 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:13.115 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:34:13.115 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-03-25T16:34:13.115 INFO:teuthology.orchestra.run.smithi120.stdout:Install 12 Packages 2024-03-25T16:34:13.115 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:13.115 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 6.9 M 2024-03-25T16:34:13.116 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 26 M 2024-03-25T16:34:13.116 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:34:13.858 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:14.077 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:34:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:34:14.182 INFO:teuthology.orchestra.run.smithi120.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 69 kB/s | 31 kB 00:00 2024-03-25T16:34:14.205 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:34:14.244 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:34:14.308 INFO:teuthology.orchestra.run.smithi120.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 26 kB/s | 15 kB 00:00 2024-03-25T16:34:14.349 INFO:teuthology.orchestra.run.smithi120.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T16:34:14.375 INFO:teuthology.orchestra.run.smithi120.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.5 MB/s | 238 kB 00:00 2024-03-25T16:34:14.423 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:34:14.459 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:34:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:34:14.584 INFO:teuthology.orchestra.run.smithi120.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-25T16:34:14.859 INFO:teuthology.orchestra.run.smithi120.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-25T16:34:14.909 INFO:teuthology.orchestra.run.smithi120.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T16:34:15.010 INFO:teuthology.orchestra.run.smithi120.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-25T16:34:15.118 INFO:teuthology.orchestra.run.smithi120.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T16:34:15.227 INFO:teuthology.orchestra.run.smithi120.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-25T16:34:15.243 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:34:15.359 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:34:15.369 INFO:teuthology.orchestra.run.smithi120.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.0 MB/s | 2.0 MB 00:00 2024-03-25T16:34:16.653 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:34:17.728 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:34:17.759 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:34:19.385 INFO:teuthology.orchestra.run.smithi120.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 322 kB/s | 1.8 MB 00:05 2024-03-25T16:34:19.390 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:19.390 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T16:34:19.410 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:34:19.428 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:34:19.428 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:34:19.609 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:34:19.609 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:34:20.113 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:34:20.202 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:34:20.246 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:34:20.292 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:34:20.301 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:34:20.301 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:34:20.301 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:34:20.301 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:34:20.301 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:34:20.301 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:34:20.302 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:34:20.302 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:34:20.302 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:34:20.302 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:34:20.302 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:34:20.302 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:34:20.323 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:34:20.431 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:34:20.556 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:34:20.649 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:34:20.695 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:34:20.724 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:34:20.738 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:34:20.738 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:20.738 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:20.738 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:20.738 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:34:20.738 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:20.739 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:20.755 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:34:20.994 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:34:21.030 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:34:21.094 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:34:21.671 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:21.824 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:34:21.824 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:34:21.825 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:34:21.826 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:34:21.828 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout:Install 13 Packages 2024-03-25T16:34:21.829 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:21.830 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 2.0 M 2024-03-25T16:34:21.830 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 26 M 2024-03-25T16:34:21.830 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:22.009 INFO:teuthology.orchestra.run.smithi042.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 172 kB/s | 18 kB 00:00 2024-03-25T16:34:22.059 INFO:teuthology.orchestra.run.smithi042.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 157 kB/s | 24 kB 00:00 2024-03-25T16:34:22.102 INFO:teuthology.orchestra.run.smithi042.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T16:34:22.127 INFO:teuthology.orchestra.run.smithi042.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T16:34:22.152 INFO:teuthology.orchestra.run.smithi042.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:34:22.177 INFO:teuthology.orchestra.run.smithi042.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:34:22.186 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:22.187 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:22.244 INFO:teuthology.orchestra.run.smithi042.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 225 kB/s | 15 kB 00:00 2024-03-25T16:34:22.269 INFO:teuthology.orchestra.run.smithi042.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 277 kB/s | 32 kB 00:00 2024-03-25T16:34:22.353 INFO:teuthology.orchestra.run.smithi042.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.8 MB/s | 203 kB 00:00 2024-03-25T16:34:22.395 INFO:teuthology.orchestra.run.smithi042.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-25T16:34:22.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:34:22.438 INFO:teuthology.orchestra.run.smithi042.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-03-25T16:34:22.496 INFO:teuthology.orchestra.run.smithi042.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-03-25T16:34:22.579 INFO:teuthology.orchestra.run.smithi042.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 77 kB/s | 52 kB 00:00 2024-03-25T16:34:22.581 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:22.581 INFO:teuthology.orchestra.run.smithi042.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-25T16:34:22.582 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:22.596 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:22.596 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:22.691 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:22.693 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:22.990 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:23.139 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:34:23.140 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:34:23.142 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-25T16:34:23.142 INFO:teuthology.orchestra.run.smithi120.stdout:Install 12 Packages 2024-03-25T16:34:23.142 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:23.142 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 37 M 2024-03-25T16:34:23.142 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 209 M 2024-03-25T16:34:23.143 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:34:23.218 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:23.351 INFO:teuthology.orchestra.run.smithi120.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 300 kB/s | 32 kB 00:00 2024-03-25T16:34:23.391 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:34:23.401 INFO:teuthology.orchestra.run.smithi120.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 638 kB/s | 17 kB 00:00 2024-03-25T16:34:23.427 INFO:teuthology.orchestra.run.smithi120.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:34:23.448 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:34:23.503 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:34:23.540 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:34:23.582 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:34:23.594 INFO:teuthology.orchestra.run.smithi120.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-25T16:34:23.678 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:34:23.736 INFO:teuthology.orchestra.run.smithi120.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T16:34:23.786 INFO:teuthology.orchestra.run.smithi120.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2024-03-25T16:34:23.787 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:34:23.811 INFO:teuthology.orchestra.run.smithi120.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-25T16:34:23.841 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:34:24.128 INFO:teuthology.orchestra.run.smithi120.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-25T16:34:24.320 INFO:teuthology.orchestra.run.smithi120.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi120.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 195 kB 00:00 2024-03-25T16:34:24.496 INFO:teuthology.orchestra.run.smithi120.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-25T16:34:24.598 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:34:24.638 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:34:24.680 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:34:24.729 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:34:24.758 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:34:25.531 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:34:25.531 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:34:25.531 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:34:25.531 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:34:25.531 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:34:25.531 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:34:25.531 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:34:25.532 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:34:25.532 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:34:25.532 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:34:25.532 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:34:25.532 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:34:25.532 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:34:25.893 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:34:25.894 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:34:25.895 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:34:25.895 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:34:25.895 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:34:25.895 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:25.895 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:26.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:34:26.713 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:26.870 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:34:26.871 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:26.872 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:26.872 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T16:34:26.872 INFO:teuthology.orchestra.run.smithi042.stdout:Install 6 Packages 2024-03-25T16:34:26.872 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:26.872 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 6.8 M 2024-03-25T16:34:26.872 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 33 M 2024-03-25T16:34:26.872 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:27.514 INFO:teuthology.orchestra.run.smithi042.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:34:27.606 INFO:teuthology.orchestra.run.smithi042.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 597 kB/s | 248 kB 00:00 2024-03-25T16:34:27.823 INFO:teuthology.orchestra.run.smithi042.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2024-03-25T16:34:27.848 INFO:teuthology.orchestra.run.smithi042.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-25T16:34:27.882 INFO:teuthology.orchestra.run.smithi042.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 870 kB/s | 51 kB 00:00 2024-03-25T16:34:28.065 INFO:teuthology.orchestra.run.smithi042.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 158 kB/s | 138 kB 00:00 2024-03-25T16:34:28.069 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:28.069 INFO:teuthology.orchestra.run.smithi042.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-25T16:34:28.082 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:28.093 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:28.093 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:28.185 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:28.186 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:28.480 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:28.647 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:34:28.701 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:34:29.273 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:34:29.365 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:34:29.434 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:34:29.462 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:34:30.258 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:34:30.258 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:34:30.258 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:34:30.258 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:34:30.258 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:34:30.258 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:30.551 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:30.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-fuse 2024-03-25T16:34:31.378 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:31.534 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:31.535 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:31.536 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:31.536 INFO:teuthology.orchestra.run.smithi042.stdout:Install 2 Packages 2024-03-25T16:34:31.536 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:31.536 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 964 k 2024-03-25T16:34:31.536 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 2.8 M 2024-03-25T16:34:31.536 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:31.750 INFO:teuthology.orchestra.run.smithi042.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 368 kB/s | 79 kB 00:00 2024-03-25T16:34:34.234 INFO:teuthology.orchestra.run.smithi042.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 328 kB/s | 885 kB 00:02 2024-03-25T16:34:34.235 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:34.236 INFO:teuthology.orchestra.run.smithi042.stdout:Total 357 kB/s | 964 kB 00:02 2024-03-25T16:34:34.236 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:34.260 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:34.260 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:34.341 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:34.341 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:34.525 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:34:34.727 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:34:35.433 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:34:35.434 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:34:35.732 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:34:35.732 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:35.733 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:35.733 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:34:35.733 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:35.733 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:35.988 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-volume 2024-03-25T16:34:36.570 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:36.726 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:36.726 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout:Install 2 Packages 2024-03-25T16:34:36.727 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:36.728 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 332 k 2024-03-25T16:34:36.728 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 1.5 M 2024-03-25T16:34:36.728 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:36.874 INFO:teuthology.orchestra.run.smithi042.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 472 kB/s | 69 kB 00:00 2024-03-25T16:34:37.799 INFO:teuthology.orchestra.run.smithi042.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 246 kB/s | 264 kB 00:01 2024-03-25T16:34:37.800 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:37.801 INFO:teuthology.orchestra.run.smithi042.stdout:Total 310 kB/s | 332 kB 00:01 2024-03-25T16:34:37.801 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:37.811 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:37.811 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:37.854 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:37.855 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:38.012 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:34:38.218 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:34:38.233 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:34:38.233 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:38.233 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:34:38.233 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:38.959 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:34:39.282 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:34:39.282 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:39.282 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:39.282 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:39.282 INFO:teuthology.orchestra.run.smithi042.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:34:39.282 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:39.282 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:39.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librados-devel 2024-03-25T16:34:40.220 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:40.375 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:40.376 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:34:40.377 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:40.377 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 127 k 2024-03-25T16:34:40.377 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 456 k 2024-03-25T16:34:40.377 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:41.605 INFO:teuthology.orchestra.run.smithi042.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 103 kB/s | 127 kB 00:01 2024-03-25T16:34:41.605 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:41.606 INFO:teuthology.orchestra.run.smithi042.stdout:Total 103 kB/s | 127 kB 00:01 2024-03-25T16:34:41.606 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:41.613 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:41.613 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:41.675 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:41.676 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:41.832 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:41.931 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:42.648 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:42.648 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:42.648 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:42.648 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:42.648 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:42.648 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:42.909 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install libcephfs2 2024-03-25T16:34:43.489 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:43.590 INFO:teuthology.orchestra.run.smithi042.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:43.646 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:43.647 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:34:43.647 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:43.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install libcephfs-devel 2024-03-25T16:34:43.985 INFO:teuthology.orchestra.run.smithi120.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 365 kB/s | 7.4 MB 00:20 2024-03-25T16:34:43.987 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:43.988 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.8 MB/s | 37 MB 00:20 2024-03-25T16:34:44.030 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:34:44.056 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:34:44.056 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:34:44.309 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:44.415 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:34:44.415 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:34:44.462 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:44.463 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:44.463 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T16:34:44.463 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:44.463 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 31 k 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 140 k 2024-03-25T16:34:44.464 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:45.041 INFO:teuthology.orchestra.run.smithi042.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 54 kB/s | 31 kB 00:00 2024-03-25T16:34:45.042 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:45.042 INFO:teuthology.orchestra.run.smithi042.stdout:Total 54 kB/s | 31 kB 00:00 2024-03-25T16:34:45.042 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:45.046 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:45.046 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:45.073 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:45.073 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:45.119 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:34:45.183 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:34:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:45.214 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:34:45.306 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:34:45.309 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:45.359 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:34:45.536 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:34:45.567 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:34:45.653 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:34:45.672 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:45.939 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:45.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:45.939 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:45.939 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:45.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:45.939 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:46.098 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librados2 2024-03-25T16:34:46.296 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:34:46.413 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:34:46.678 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:46.778 INFO:teuthology.orchestra.run.smithi042.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:46.833 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:46.834 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:34:46.834 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:46.890 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librbd1 2024-03-25T16:34:47.474 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:47.574 INFO:teuthology.orchestra.run.smithi042.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:47.629 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:47.630 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:34:47.630 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:47.630 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:34:47.679 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rados 2024-03-25T16:34:48.254 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:01 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:48.353 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:48.409 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:48.410 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:34:48.410 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:48.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rgw 2024-03-25T16:34:48.656 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:34:48.685 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:34:49.056 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:49.157 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:49.213 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:49.214 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:34:49.214 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:49.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-cephfs 2024-03-25T16:34:49.847 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:49.947 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:50.014 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:34:50.014 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:50.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rbd 2024-03-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:50.745 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:50.799 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:50.800 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T16:34:50.800 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:50.852 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-fuse 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:34:51.306 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:34:51.307 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:34:51.427 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:51.582 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:51.583 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:51.583 INFO:teuthology.orchestra.run.smithi042.stdout: Package Architecture Version Repository Size 2024-03-25T16:34:51.583 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:51.583 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 85 k 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 231 k 2024-03-25T16:34:51.584 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:34:51.796 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:34:51.797 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:34:51.797 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:34:51.797 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:34:51.797 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:34:51.797 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:34:51.797 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:51.797 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:52.167 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:34:52.420 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 102 kB/s | 85 kB 00:00 2024-03-25T16:34:52.420 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:52.421 INFO:teuthology.orchestra.run.smithi042.stdout:Total 102 kB/s | 85 kB 00:00 2024-03-25T16:34:52.421 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:34:52.430 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:34:52.430 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:34:52.492 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:34:52.493 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:34:52.651 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:34:52.732 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:52.762 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:52.881 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:52.882 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T16:34:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:34:52.882 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:34:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout:Install 13 Packages 2024-03-25T16:34:52.884 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:52.885 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 2.0 M 2024-03-25T16:34:52.885 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 26 M 2024-03-25T16:34:52.885 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:34:53.120 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:53.403 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:53.403 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:53.403 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:34:53.403 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:53.403 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:53.403 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:34:53.446 INFO:teuthology.orchestra.run.smithi120.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 150 kB/s | 24 kB 00:00 2024-03-25T16:34:53.463 INFO:teuthology.orchestra.run.smithi120.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 101 kB/s | 18 kB 00:00 2024-03-25T16:34:53.654 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-mirror 2024-03-25T16:34:53.680 INFO:teuthology.orchestra.run.smithi120.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 747 kB/s | 174 kB 00:00 2024-03-25T16:34:53.747 INFO:teuthology.orchestra.run.smithi120.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 466 kB/s | 132 kB 00:00 2024-03-25T16:34:53.780 INFO:teuthology.orchestra.run.smithi120.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 211 kB 00:00 2024-03-25T16:34:53.797 INFO:teuthology.orchestra.run.smithi120.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-25T16:34:54.048 INFO:teuthology.orchestra.run.smithi120.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 68 kB/s | 52 kB 00:00 2024-03-25T16:34:54.173 INFO:teuthology.orchestra.run.smithi120.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 40 kB/s | 15 kB 00:00 2024-03-25T16:34:54.190 INFO:teuthology.orchestra.run.smithi120.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 79 kB/s | 32 kB 00:00 2024-03-25T16:34:54.232 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:34:54.365 INFO:teuthology.orchestra.run.smithi120.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-25T16:34:54.387 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:34:54.388 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:54.388 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 3.1 M 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 13 M 2024-03-25T16:34:54.389 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:34:54.507 INFO:teuthology.orchestra.run.smithi120.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 633 kB/s | 90 kB 00:00 2024-03-25T16:34:54.532 INFO:teuthology.orchestra.run.smithi120.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 418 kB/s | 203 kB 00:00 2024-03-25T16:34:54.599 INFO:teuthology.orchestra.run.smithi120.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-03-25T16:34:54.603 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:54.603 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-25T16:34:54.604 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:34:54.617 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:34:54.617 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:34:54.708 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:34:54.708 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:34:55.221 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:34:55.380 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:34:55.434 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:34:55.489 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:34:55.518 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:34:55.560 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:34:55.661 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:34:55.771 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:34:55.819 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:34:56.537 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:34:56.584 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:34:56.618 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:34:56.702 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:34:56.731 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:34:57.560 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:34:57.560 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:34:57.560 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:34:57.560 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:34:57.561 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:34:57.924 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:34:57.925 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:34:57.926 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:34:57.926 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:34:57.926 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:34:57.926 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:34:57.926 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:34:57.926 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:57.926 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:34:58.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:34:58.749 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:34:58.900 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:34:58.901 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T16:34:58.901 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:34:58.901 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T16:34:58.901 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:34:58.901 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:34:58.901 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2024-03-25T16:34:58.902 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:34:58.903 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 6.8 M 2024-03-25T16:34:58.903 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 33 M 2024-03-25T16:34:58.903 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:34:59.580 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T16:34:59.605 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 832 kB/s | 248 kB 00:00 2024-03-25T16:34:59.697 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 627 kB/s | 58 kB 00:00 2024-03-25T16:34:59.722 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-03-25T16:34:59.748 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.2 MB/s | 548 kB 00:00 2024-03-25T16:35:00.191 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 156 kB/s | 138 kB 00:00 2024-03-25T16:35:00.192 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:00.192 INFO:teuthology.orchestra.run.smithi120.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-25T16:35:00.208 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:00.219 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:00.219 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:00.309 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:00.310 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:00.602 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:00.762 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:35:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:35:01.330 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:35:01.410 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:35:01.480 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:35:01.504 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:35:02.364 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:35:02.364 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:35:02.364 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:35:02.364 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:35:02.364 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:35:02.364 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:35:02.672 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:35:02.672 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:02.672 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:02.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:02.673 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:35:02.673 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:35:02.673 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:35:02.673 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:35:02.673 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:35:02.673 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:02.673 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:02.920 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-fuse 2024-03-25T16:35:03.485 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:03.635 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:03.636 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:03.637 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:03.637 INFO:teuthology.orchestra.run.smithi120.stdout:Install 2 Packages 2024-03-25T16:35:03.637 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:03.637 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 964 k 2024-03-25T16:35:03.637 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.8 M 2024-03-25T16:35:03.637 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:35:03.742 INFO:teuthology.orchestra.run.smithi120.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 756 kB/s | 79 kB 00:00 2024-03-25T16:35:05.230 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 292 kB/s | 3.1 MB 00:10 2024-03-25T16:35:05.231 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:05.231 INFO:teuthology.orchestra.run.smithi042.stdout:Total 292 kB/s | 3.1 MB 00:10 2024-03-25T16:35:05.232 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:35:05.244 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:35:05.244 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:35:05.323 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:35:05.324 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:35:05.775 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:35:05.831 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:05.855 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:05.855 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:05.856 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:35:05.856 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:35:05.856 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:35:05.856 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:06.986 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:06.986 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:06.987 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:35:06.987 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:06.987 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:06.987 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:35:07.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-nbd 2024-03-25T16:35:07.525 INFO:teuthology.orchestra.run.smithi120.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 228 kB/s | 885 kB 00:03 2024-03-25T16:35:07.526 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:07.527 INFO:teuthology.orchestra.run.smithi120.stdout:Total 248 kB/s | 964 kB 00:03 2024-03-25T16:35:07.527 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:07.549 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:07.550 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:07.626 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:07.626 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:07.809 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:07.809 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:35:07.943 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:35:07.963 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:35:07.963 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:35:07.963 INFO:teuthology.orchestra.run.smithi042.stdout: Package Architecture Version Repository Size 2024-03-25T16:35:07.963 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:07.964 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 171 k 2024-03-25T16:35:07.965 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 494 k 2024-03-25T16:35:07.965 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:35:08.017 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:35:08.677 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:35:08.677 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:35:08.995 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:35:08.996 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:08.996 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:08.996 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:35:08.996 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:08.996 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:09.133 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 147 kB/s | 171 kB 00:01 2024-03-25T16:35:09.134 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:09.134 INFO:teuthology.orchestra.run.smithi042.stdout:Total 147 kB/s | 171 kB 00:01 2024-03-25T16:35:09.134 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:35:09.144 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:35:09.144 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:35:09.207 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:35:09.208 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:35:09.263 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-volume 2024-03-25T16:35:09.363 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:35:09.468 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:09.840 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:21 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:09.876 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:09.990 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:09.991 INFO:teuthology.orchestra.run.smithi120.stdout:Install 2 Packages 2024-03-25T16:35:09.992 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:09.992 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 332 k 2024-03-25T16:35:09.992 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.5 M 2024-03-25T16:35:09.992 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:35:10.106 INFO:teuthology.orchestra.run.smithi120.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 606 kB/s | 69 kB 00:00 2024-03-25T16:35:10.145 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:10.145 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:10.145 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:35:10.145 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:10.145 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:10.145 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:35:10.365 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:35:10.941 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:06:23 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:35:11.098 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:35:11.098 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout:Install 2 Packages 2024-03-25T16:35:11.099 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:35:11.100 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 91 M 2024-03-25T16:35:11.100 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 498 M 2024-03-25T16:35:11.100 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:35:12.039 INFO:teuthology.orchestra.run.smithi120.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 129 kB/s | 264 kB 00:02 2024-03-25T16:35:12.041 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:12.041 INFO:teuthology.orchestra.run.smithi120.stdout:Total 162 kB/s | 332 kB 00:02 2024-03-25T16:35:12.041 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:12.051 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:12.051 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:12.093 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:12.094 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:12.247 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:12.411 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:35:12.442 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:35:12.458 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:35:12.458 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:12.458 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:35:12.459 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:13.182 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:35:13.502 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:35:13.503 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:13.503 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:13.503 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:13.503 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:35:13.503 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:13.503 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:13.811 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados-devel 2024-03-25T16:35:14.382 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:14.529 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:14.530 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:35:14.531 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:14.531 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 127 k 2024-03-25T16:35:14.531 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 456 k 2024-03-25T16:35:14.531 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:35:15.947 INFO:teuthology.orchestra.run.smithi120.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 89 kB/s | 127 kB 00:01 2024-03-25T16:35:15.948 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:15.948 INFO:teuthology.orchestra.run.smithi120.stdout:Total 89 kB/s | 127 kB 00:01 2024-03-25T16:35:15.948 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:15.954 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:15.955 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:16.014 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:16.015 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:16.158 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:16.254 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:16.682 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:16.987 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:16.987 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:16.987 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:16.987 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:16.987 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:16.988 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:17.236 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs2 2024-03-25T16:35:17.809 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:29 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:17.908 INFO:teuthology.orchestra.run.smithi120.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:17.964 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:17.965 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:35:17.965 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:18.033 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs-devel 2024-03-25T16:35:18.612 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:18.770 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:35:18.771 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:18.772 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 31 k 2024-03-25T16:35:18.772 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 140 k 2024-03-25T16:35:18.772 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:35:19.305 INFO:teuthology.orchestra.run.smithi120.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T16:35:19.306 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:19.306 INFO:teuthology.orchestra.run.smithi120.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T16:35:19.306 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:19.308 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:19.308 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:19.329 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:19.330 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:19.458 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:19.559 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:19.964 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:20.227 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:20.227 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:20.227 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:20.227 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:20.227 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:20.227 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:20.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados2 2024-03-25T16:35:20.936 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:32 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:21.032 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:21.083 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:21.084 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:35:21.084 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:21.138 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librbd1 2024-03-25T16:35:21.699 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:33 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:21.797 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:21.848 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:21.850 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:35:21.850 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:21.901 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rados 2024-03-25T16:35:22.460 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:22.555 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:22.606 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:22.607 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:35:22.607 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:22.659 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rgw 2024-03-25T16:35:23.224 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:23.320 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:23.372 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:23.373 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:35:23.373 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:23.426 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-cephfs 2024-03-25T16:35:23.991 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:24.087 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:24.138 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:24.139 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:35:24.139 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:24.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rbd 2024-03-25T16:35:24.744 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:24.841 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:24.892 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:24.893 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T16:35:24.893 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:24.942 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-fuse 2024-03-25T16:35:25.507 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:37 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:25.655 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 85 k 2024-03-25T16:35:25.656 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 231 k 2024-03-25T16:35:25.657 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:35:26.564 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 94 kB/s | 85 kB 00:00 2024-03-25T16:35:26.565 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:26.565 INFO:teuthology.orchestra.run.smithi120.stdout:Total 94 kB/s | 85 kB 00:00 2024-03-25T16:35:26.566 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:26.574 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:26.574 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:26.633 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:26.633 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:26.774 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:26.862 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:27.284 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:27.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:27.593 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:27.594 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:27.594 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:27.594 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:27.594 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:27.824 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-mirror 2024-03-25T16:35:28.399 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:40 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:28.554 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:28.555 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:28.556 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:35:28.556 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:28.556 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 3.1 M 2024-03-25T16:35:28.556 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 13 M 2024-03-25T16:35:28.556 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:35:40.834 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 258 kB/s | 3.1 MB 00:12 2024-03-25T16:35:40.835 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:40.835 INFO:teuthology.orchestra.run.smithi120.stdout:Total 257 kB/s | 3.1 MB 00:12 2024-03-25T16:35:40.835 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:40.846 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:40.846 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:40.921 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:40.922 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:41.347 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:41.410 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:41.434 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:41.434 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:41.434 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:35:41.434 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:35:41.434 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:35:41.434 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:42.540 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:42.540 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:42.540 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:42.540 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:42.540 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:42.540 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:42.771 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-nbd 2024-03-25T16:35:43.353 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:55 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:43.503 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:43.504 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:43.504 INFO:teuthology.orchestra.run.smithi120.stdout: Package Architecture Version Repository Size 2024-03-25T16:35:43.504 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 171 k 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 494 k 2024-03-25T16:35:43.505 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:35:44.496 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 173 kB/s | 171 kB 00:00 2024-03-25T16:35:44.497 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:44.498 INFO:teuthology.orchestra.run.smithi120.stdout:Total 173 kB/s | 171 kB 00:00 2024-03-25T16:35:44.498 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:35:44.507 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:35:44.507 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:35:44.565 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:35:44.566 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:35:44.718 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:35:44.828 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:45.304 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:35:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:45.598 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:35:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:45.599 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:35:45.824 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:35:46.389 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:58 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:35:46.542 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:35:46.542 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:46.542 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T16:35:46.542 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:46.542 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout:Install 2 Packages 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 91 M 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 498 M 2024-03-25T16:35:46.543 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T16:36:42.478 INFO:teuthology.orchestra.run.smithi042.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 252 kB/s | 22 MB 01:31 2024-03-25T16:37:00.641 INFO:teuthology.orchestra.run.smithi120.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 310 kB/s | 22 MB 01:14 2024-03-25T16:38:15.127 INFO:teuthology.orchestra.run.smithi042.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 379 kB/s | 68 MB 03:04 2024-03-25T16:38:15.128 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:15.128 INFO:teuthology.orchestra.run.smithi042.stdout:Total 504 kB/s | 91 MB 03:04 2024-03-25T16:38:15.128 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T16:38:15.131 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T16:38:15.132 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T16:38:15.735 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T16:38:15.736 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T16:38:19.012 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T16:38:26.532 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:38:26.605 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:38:29.976 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:38:29.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:38:30.243 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:30.243 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:38:30.243 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T16:38:30.243 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:30.243 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:30.243 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:38:30.243 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T16:38:30.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:38:31.039 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:09:44 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T16:38:31.194 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T16:38:31.195 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:38:31.195 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T16:38:31.195 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 416 M 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 1.7 G 2024-03-25T16:38:31.196 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T16:39:49.120 INFO:teuthology.orchestra.run.smithi120.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 288 kB/s | 68 MB 04:02 2024-03-25T16:39:49.121 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:49.121 INFO:teuthology.orchestra.run.smithi120.stdout:Total 382 kB/s | 91 MB 04:02 2024-03-25T16:39:49.121 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T16:39:49.126 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T16:39:49.126 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T16:39:49.727 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T16:39:49.727 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T16:39:52.824 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T16:39:59.963 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:40:00.036 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:40:03.493 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:40:03.494 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:40:03.764 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:40:03.765 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:40:03.765 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T16:40:03.765 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:03.765 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:03.765 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:40:03.765 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T16:40:03.988 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:40:04.579 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:11:16 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T16:40:04.731 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T16:40:04.732 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:40:04.732 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T16:40:04.732 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:40:04.732 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 416 M 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.7 G 2024-03-25T16:40:04.733 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:01:10.540 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 313 kB/s | 416 MB 22:39 2024-03-25T17:01:10.540 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:01:10.541 INFO:teuthology.orchestra.run.smithi042.stdout:Total 313 kB/s | 416 MB 22:39 2024-03-25T17:01:10.541 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T17:01:10.544 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T17:01:10.545 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T17:01:13.132 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T17:01:13.132 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T17:01:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T17:01:57.905 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:02.977 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:03.251 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:03.251 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:03.251 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T17:02:03.251 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:02:03.251 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:03.252 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:02:03.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:02:04.073 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:33:17 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:02:04.229 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:04.230 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 1.4 M 2024-03-25T17:02:04.231 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 6.4 M 2024-03-25T17:02:04.231 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T17:02:06.180 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 749 kB/s | 1.4 MB 00:01 2024-03-25T17:02:06.181 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:02:06.181 INFO:teuthology.orchestra.run.smithi042.stdout:Total 749 kB/s | 1.4 MB 00:01 2024-03-25T17:02:06.182 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T17:02:06.186 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T17:02:06.186 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T17:02:06.233 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T17:02:06.233 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T17:02:06.500 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T17:02:06.599 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:06.991 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:07.270 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:07.270 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:07.270 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T17:02:07.271 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:02:07.271 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:07.271 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:02:07.452 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:02:08.026 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:33:21 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:02:08.185 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:02:08.186 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:02:08.186 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T17:02:08.186 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 176 M 2024-03-25T17:02:08.187 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 753 M 2024-03-25T17:02:08.188 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T17:06:17.621 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 271 kB/s | 416 MB 26:12 2024-03-25T17:06:17.621 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:06:17.622 INFO:teuthology.orchestra.run.smithi120.stdout:Total 271 kB/s | 416 MB 26:12 2024-03-25T17:06:17.622 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T17:06:17.625 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T17:06:17.625 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T17:06:20.088 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T17:06:20.088 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T17:07:02.281 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T17:07:02.381 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:07:10.701 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:07:10.956 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:07:10.957 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:10.957 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T17:07:10.957 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:07:10.957 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:10.957 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:07:11.198 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:07:11.762 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:38:23 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:07:11.909 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:======================================================================================== 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.4 M 2024-03-25T17:07:11.910 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 6.4 M 2024-03-25T17:07:11.911 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:07:16.644 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 309 kB/s | 1.4 MB 00:04 2024-03-25T17:07:16.645 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:07:16.645 INFO:teuthology.orchestra.run.smithi120.stdout:Total 309 kB/s | 1.4 MB 00:04 2024-03-25T17:07:16.645 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T17:07:16.649 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T17:07:16.649 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T17:07:16.693 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T17:07:16.694 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T17:07:16.950 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T17:07:17.042 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:07:17.442 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:07:17.740 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:07:17.740 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:17.740 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T17:07:17.740 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:07:17.740 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:17.740 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:07:17.916 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:07:18.477 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:38:30 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T17:07:18.625 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:07:18.626 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 176 M 2024-03-25T17:07:18.626 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 753 M 2024-03-25T17:07:18.626 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:12:14.321 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 297 kB/s | 176 MB 10:06 2024-03-25T17:12:14.322 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:12:14.322 INFO:teuthology.orchestra.run.smithi042.stdout:Total 297 kB/s | 176 MB 10:06 2024-03-25T17:12:14.322 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T17:12:14.324 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T17:12:14.324 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T17:12:15.433 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T17:12:15.433 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T17:12:34.462 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T17:12:34.553 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:12:39.247 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:12:39.504 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:12:39.505 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:12:39.505 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T17:12:39.505 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:12:39.505 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:12:39.505 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:12:39.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:12:40.300 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:43:53 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:12:40.455 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T17:12:40.456 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:12:40.457 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 672 M 2024-03-25T17:12:40.457 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 3.1 G 2024-03-25T17:12:40.457 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T17:17:38.072 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 290 kB/s | 176 MB 10:19 2024-03-25T17:17:38.095 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:17:38.095 INFO:teuthology.orchestra.run.smithi120.stdout:Total 290 kB/s | 176 MB 10:19 2024-03-25T17:17:38.095 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T17:17:38.096 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T17:17:38.096 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T17:17:39.137 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T17:17:39.138 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T17:17:57.290 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T17:17:57.484 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:18:02.392 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:18:02.661 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:18:02.661 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:18:02.661 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T17:18:02.661 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:18:02.661 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:18:02.661 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:18:02.848 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:18:03.411 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:49:15 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:18:03.561 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T17:18:03.562 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:18:03.563 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 672 M 2024-03-25T17:18:03.563 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 3.1 G 2024-03-25T17:18:03.563 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:39:23.599 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 538 kB/s | 672 MB 21:20 2024-03-25T17:39:23.624 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:23.624 INFO:teuthology.orchestra.run.smithi120.stdout:Total 538 kB/s | 672 MB 21:20 2024-03-25T17:39:23.624 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T17:39:23.624 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T17:39:23.625 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T17:39:27.560 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T17:39:27.560 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T17:40:38.375 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T17:40:38.617 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:40:47.856 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:40:48.121 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:40:48.121 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:40:48.121 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T17:40:48.121 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:48.121 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:40:48.121 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:40:48.400 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:40:48.965 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:12:00 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:40:49.062 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:40:49.114 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:40:49.115 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2024-03-25T17:40:49.115 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:40:49.195 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:40:49.764 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:12:01 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:40:49.913 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:40:49.913 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:40:49.913 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T17:40:49.913 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:40:49.914 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T17:40:49.914 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:40:49.914 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:40:49.914 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T17:40:49.914 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:40:49.914 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T17:40:49.914 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:40:49.915 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 20 M 2024-03-25T17:40:49.915 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 103 M 2024-03-25T17:40:49.915 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:41:36.874 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 430 kB/s | 20 MB 00:46 2024-03-25T17:41:36.875 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:36.875 INFO:teuthology.orchestra.run.smithi120.stdout:Total 430 kB/s | 20 MB 00:46 2024-03-25T17:41:36.875 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T17:41:36.879 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T17:41:36.879 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T17:41:37.029 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T17:41:37.030 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T17:41:39.261 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T17:41:39.355 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:40.322 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:40.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:40.586 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:41:40.586 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T17:41:40.586 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:40.586 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:41:40.586 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:41:40.744 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:41:41.330 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:12:53 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:41:41.488 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:41:41.489 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:41:41.489 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T17:41:41.489 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 48 M 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 173 M 2024-03-25T17:41:41.490 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:43:04.896 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 377 kB/s | 672 MB 30:24 2024-03-25T17:43:04.896 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:04.897 INFO:teuthology.orchestra.run.smithi042.stdout:Total 377 kB/s | 672 MB 30:24 2024-03-25T17:43:04.897 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T17:43:04.901 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T17:43:04.901 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T17:43:09.053 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T17:43:09.053 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T17:43:40.303 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 411 kB/s | 48 MB 01:58 2024-03-25T17:43:40.304 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:40.304 INFO:teuthology.orchestra.run.smithi120.stdout:Total 411 kB/s | 48 MB 01:58 2024-03-25T17:43:40.304 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T17:43:40.307 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T17:43:40.308 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T17:43:40.619 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T17:43:40.620 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T17:43:45.339 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T17:43:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:46.956 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:47.233 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:43:47.234 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T17:43:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:47.234 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:43:47.234 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:43:47.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:43:47.957 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:14:59 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:43:48.106 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:43:48.107 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:43:48.107 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T17:43:48.107 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:43:48.107 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T17:43:48.107 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:43:48.107 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:43:48.107 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T17:43:48.108 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:43:48.108 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T17:43:48.108 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:43:48.108 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 99 M 2024-03-25T17:43:48.108 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 449 M 2024-03-25T17:43:48.108 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:44:24.091 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T17:44:24.400 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:44:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:44:34.142 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:44:34.142 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:44:34.142 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T17:44:34.142 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:34.142 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:44:34.143 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:44:34.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:44:34.985 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:15:47 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:44:35.084 INFO:teuthology.orchestra.run.smithi042.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:44:35.139 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:44:35.140 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-25T17:44:35.140 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:44:35.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:44:35.776 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:15:48 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:44:35.933 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:44:35.934 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:44:35.934 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T17:44:35.934 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:44:35.934 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T17:44:35.934 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:44:35.934 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:44:35.935 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T17:44:35.935 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:44:35.935 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T17:44:35.935 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:44:35.935 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 20 M 2024-03-25T17:44:35.935 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 103 M 2024-03-25T17:44:35.935 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T17:45:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 315 kB/s | 20 MB 01:04 2024-03-25T17:45:40.004 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:40.004 INFO:teuthology.orchestra.run.smithi042.stdout:Total 315 kB/s | 20 MB 01:04 2024-03-25T17:45:40.004 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T17:45:40.007 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T17:45:40.007 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T17:45:40.164 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T17:45:40.164 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T17:45:42.520 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T17:45:42.636 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:43.569 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:43.829 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:43.829 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:45:43.829 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T17:45:43.829 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:43.829 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:45:43.829 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:45:43.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:45:44.571 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:16:57 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:45:44.724 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:45:44.725 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:45:44.725 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T17:45:44.725 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:45:44.725 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T17:45:44.725 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:45:44.725 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:45:44.726 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T17:45:44.726 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:45:44.726 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T17:45:44.726 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:45:44.726 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 48 M 2024-03-25T17:45:44.726 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 173 M 2024-03-25T17:45:44.726 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T17:48:08.038 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 388 kB/s | 99 MB 04:19 2024-03-25T17:48:08.039 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:08.039 INFO:teuthology.orchestra.run.smithi120.stdout:Total 388 kB/s | 99 MB 04:19 2024-03-25T17:48:08.039 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T17:48:08.042 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T17:48:08.042 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T17:48:08.651 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T17:48:08.651 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T17:48:19.011 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:22.230 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:22.497 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:22.497 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:48:22.497 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T17:48:22.497 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:22.497 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:48:22.497 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T17:48:22.655 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:48:23.215 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:19:35 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T17:48:23.361 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T17:48:23.362 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:48:23.362 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 287 M 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.2 G 2024-03-25T17:48:23.363 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T17:48:39.012 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 280 kB/s | 48 MB 02:54 2024-03-25T17:48:39.013 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:39.013 INFO:teuthology.orchestra.run.smithi042.stdout:Total 280 kB/s | 48 MB 02:54 2024-03-25T17:48:39.013 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T17:48:39.017 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T17:48:39.017 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T17:48:39.345 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T17:48:39.345 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T17:48:44.369 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T17:48:44.463 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:46.098 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:46.098 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:48:46.098 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T17:48:46.098 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:46.098 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:48:46.098 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:48:46.273 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:48:46.847 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:19:59 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:48:47.001 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:48:47.002 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:48:47.002 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T17:48:47.002 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:48:47.002 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T17:48:47.002 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:48:47.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:48:47.003 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T17:48:47.003 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:48:47.003 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T17:48:47.003 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:48:47.003 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 99 M 2024-03-25T17:48:47.003 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 449 M 2024-03-25T17:48:47.003 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T17:52:57.106 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 403 kB/s | 99 MB 04:10 2024-03-25T17:52:57.107 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:57.107 INFO:teuthology.orchestra.run.smithi042.stdout:Total 403 kB/s | 99 MB 04:10 2024-03-25T17:52:57.107 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T17:52:57.110 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T17:52:57.110 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T17:52:57.750 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T17:52:57.751 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T17:53:08.637 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T17:53:08.731 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:11.687 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:11.952 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:11.952 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:53:11.952 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T17:53:11.952 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:11.952 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:53:11.953 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T17:53:12.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:53:12.704 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:24:25 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T17:53:12.861 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T17:53:12.862 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:53:12.862 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T17:53:12.862 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:53:12.862 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T17:53:12.862 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:53:12.862 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:53:12.863 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T17:53:12.863 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T17:53:12.863 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T17:53:12.863 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T17:53:12.863 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 287 M 2024-03-25T17:53:12.863 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 1.2 G 2024-03-25T17:53:12.863 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:04:09.839 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 310 kB/s | 287 MB 15:46 2024-03-25T18:04:09.839 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:09.839 INFO:teuthology.orchestra.run.smithi120.stdout:Total 310 kB/s | 287 MB 15:46 2024-03-25T18:04:09.839 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:04:09.842 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:04:09.842 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:04:11.543 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:04:11.544 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:04:40.765 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:04:40.875 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:42.042 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:42.338 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:42.339 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:04:42.339 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:04:42.339 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:42.339 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:04:42.339 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:04:42.517 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:04:43.076 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:35:55 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:04:43.224 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:04:43.225 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:04:43.226 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 5.9 M 2024-03-25T18:04:43.226 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 28 M 2024-03-25T18:04:43.226 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:05:02.019 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 324 kB/s | 5.9 MB 00:18 2024-03-25T18:05:02.019 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:02.020 INFO:teuthology.orchestra.run.smithi120.stdout:Total 324 kB/s | 5.9 MB 00:18 2024-03-25T18:05:02.020 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:05:02.024 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:05:02.024 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:05:02.096 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:05:02.096 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:05:02.827 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:05:02.942 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:03.539 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:03.823 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:05:03.823 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:03.823 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:05:03.823 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:03.823 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:03.824 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:05:04.019 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:05:04.579 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:36:16 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:05:04.728 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:05:04.729 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:04.730 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:05:04.730 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:04.730 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 109 k 2024-03-25T18:05:04.730 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 440 k 2024-03-25T18:05:04.730 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:05:05.480 INFO:teuthology.orchestra.run.smithi120.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 145 kB/s | 109 kB 00:00 2024-03-25T18:05:05.480 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:05.481 INFO:teuthology.orchestra.run.smithi120.stdout:Total 145 kB/s | 109 kB 00:00 2024-03-25T18:05:05.481 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:05:05.484 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:05:05.484 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:05:05.519 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:05:05.519 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:05:05.632 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:05:05.719 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:06.091 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:06.373 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:06.373 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:06.373 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:05:06.373 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:06.373 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:06.373 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:05:06.554 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:05:07.118 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:36:19 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:05:07.266 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:05:07.267 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:07.268 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:05:07.268 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:07.268 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 5.4 M 2024-03-25T18:05:07.268 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 25 M 2024-03-25T18:05:07.268 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:05:23.781 INFO:teuthology.orchestra.run.smithi120.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 332 kB/s | 5.4 MB 00:16 2024-03-25T18:05:23.782 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:23.782 INFO:teuthology.orchestra.run.smithi120.stdout:Total 332 kB/s | 5.4 MB 00:16 2024-03-25T18:05:23.782 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:05:23.786 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:05:23.786 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:05:23.854 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:05:23.854 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:05:24.515 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:05:24.612 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:25.135 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:25.410 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:05:25.593 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librados2-debuginfo 2024-03-25T18:05:26.160 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:36:38 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:05:26.308 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:05:26.308 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:26.308 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:05:26.308 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:26.308 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:05:26.309 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:05:26.309 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:26.309 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:05:26.309 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:05:26.309 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:05:26.309 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:05:26.310 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 44 M 2024-03-25T18:05:26.310 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 253 M 2024-03-25T18:05:26.310 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:07:31.201 INFO:teuthology.orchestra.run.smithi120.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 363 kB/s | 44 MB 02:04 2024-03-25T18:07:31.201 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:31.202 INFO:teuthology.orchestra.run.smithi120.stdout:Total 363 kB/s | 44 MB 02:04 2024-03-25T18:07:31.202 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:07:31.204 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:07:31.205 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:07:31.503 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:07:31.503 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:07:36.547 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:07:36.657 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:38.577 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:38.858 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:38.858 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:07:38.858 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:07:38.858 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:38.858 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:07:38.858 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:07:39.048 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:07:39.610 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:38:51 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:07:39.759 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:07:39.760 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:07:39.761 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:07:39.761 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:07:39.761 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 52 M 2024-03-25T18:07:39.761 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 308 M 2024-03-25T18:07:39.761 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:08:38.927 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 317 kB/s | 287 MB 15:26 2024-03-25T18:08:38.928 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:38.928 INFO:teuthology.orchestra.run.smithi042.stdout:Total 317 kB/s | 287 MB 15:26 2024-03-25T18:08:38.928 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:08:38.931 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:08:38.931 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:08:40.720 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:08:40.721 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:09:11.843 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:09:11.958 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:09:19.748 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:09:20.031 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:09:20.031 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:20.031 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:09:20.032 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:20.032 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:20.032 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:09:20.328 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:09:20.905 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:40:33 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:09:21.059 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:09:21.060 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:21.060 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:09:21.060 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 5.9 M 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 28 M 2024-03-25T18:09:21.061 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:09:41.741 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 294 kB/s | 5.9 MB 00:20 2024-03-25T18:09:41.742 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:41.742 INFO:teuthology.orchestra.run.smithi042.stdout:Total 294 kB/s | 5.9 MB 00:20 2024-03-25T18:09:41.742 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:09:41.747 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:09:41.747 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:09:41.821 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:09:41.821 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:09:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:09:42.701 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:09:43.242 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:09:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:09:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:43.509 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:09:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:43.509 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:09:43.697 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:09:44.272 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:40:57 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:09:44.428 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:09:44.428 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:44.428 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:09:44.428 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:44.428 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 109 k 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 440 k 2024-03-25T18:09:44.429 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:09:45.127 INFO:teuthology.orchestra.run.smithi042.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 156 kB/s | 109 kB 00:00 2024-03-25T18:09:45.128 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:45.128 INFO:teuthology.orchestra.run.smithi042.stdout:Total 155 kB/s | 109 kB 00:00 2024-03-25T18:09:45.128 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:09:45.132 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:09:45.132 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:09:45.169 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:09:45.169 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:09:45.289 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:09:45.403 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:09:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:09:46.022 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:09:46.023 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:46.023 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:09:46.023 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:46.023 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:46.023 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:09:46.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:09:46.778 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:40:59 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:09:46.932 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:09:46.933 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 5.4 M 2024-03-25T18:09:46.934 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 25 M 2024-03-25T18:09:46.934 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:10:06.612 INFO:teuthology.orchestra.run.smithi042.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 279 kB/s | 5.4 MB 00:19 2024-03-25T18:10:06.613 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:06.613 INFO:teuthology.orchestra.run.smithi042.stdout:Total 278 kB/s | 5.4 MB 00:19 2024-03-25T18:10:06.614 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:10:06.618 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:10:06.618 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:10:06.689 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:10:06.689 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:10:07.390 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:10:07.505 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:10:08.170 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:10:08.439 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:10:08.439 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:10:08.440 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:10:08.440 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:08.440 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:10:08.440 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:10:08.793 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librados2-debuginfo 2024-03-25T18:10:09.368 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:41:22 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:10:09.525 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 44 M 2024-03-25T18:10:09.526 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 253 M 2024-03-25T18:10:09.527 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:10:57.326 INFO:teuthology.orchestra.run.smithi120.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 269 kB/s | 52 MB 03:17 2024-03-25T18:10:57.327 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:57.327 INFO:teuthology.orchestra.run.smithi120.stdout:Total 269 kB/s | 52 MB 03:17 2024-03-25T18:10:57.327 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:10:57.330 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:10:57.331 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:10:57.673 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:10:57.674 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:11:03.631 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:11:03.841 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:06.068 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:06.348 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:06.348 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:06.348 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:11:06.348 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:06.349 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:06.349 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:11:06.577 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:11:07.133 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:42:19 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:07.282 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:11:07.283 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:07.283 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:11:07.283 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:07.283 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 615 k 2024-03-25T18:11:07.283 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.1 M 2024-03-25T18:11:07.283 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:11:08.587 INFO:teuthology.orchestra.run.smithi120.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 472 kB/s | 615 kB 00:01 2024-03-25T18:11:08.587 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:08.588 INFO:teuthology.orchestra.run.smithi120.stdout:Total 471 kB/s | 615 kB 00:01 2024-03-25T18:11:08.588 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:11:08.592 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:11:08.592 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:11:08.632 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:11:08.632 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:11:08.803 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:11:08.911 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:09.307 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:09.599 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:09.599 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:09.599 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:11:09.599 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:09.599 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:09.599 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:11:09.804 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:11:10.360 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:42:22 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:11:10.508 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:11:10.509 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:10.509 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:11:10.509 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:10.509 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 771 k 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.7 M 2024-03-25T18:11:10.510 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:11:12.414 INFO:teuthology.orchestra.run.smithi120.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 405 kB/s | 771 kB 00:01 2024-03-25T18:11:12.415 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:12.415 INFO:teuthology.orchestra.run.smithi120.stdout:Total 404 kB/s | 771 kB 00:01 2024-03-25T18:11:12.416 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:11:12.419 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:11:12.419 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:11:12.460 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:11:12.461 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:11:12.643 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:11:12.746 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:11:13.178 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:11:13.455 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:11:13.455 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:13.455 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:11:13.455 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:13.456 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:13.456 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:11:13.681 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:11:14.244 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:42:26 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:11:14.393 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:11:14.394 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:14.394 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:11:14.394 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:14.394 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:11:14.394 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.1 M 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 4.3 M 2024-03-25T18:11:14.395 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:11:18.562 INFO:teuthology.orchestra.run.smithi120.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 276 kB/s | 1.1 MB 00:04 2024-03-25T18:11:18.562 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:18.562 INFO:teuthology.orchestra.run.smithi120.stdout:Total 276 kB/s | 1.1 MB 00:04 2024-03-25T18:11:18.562 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:11:18.564 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:11:18.564 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:11:18.607 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:11:18.608 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:11:18.832 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:11:18.949 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:19.341 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:19.621 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:19.622 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:19.622 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:11:19.622 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:19.622 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:19.622 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:11:19.839 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:11:20.399 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:42:32 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:11:20.548 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:11:20.549 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:20.549 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 337 k 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 1.0 M 2024-03-25T18:11:20.550 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:11:22.295 INFO:teuthology.orchestra.run.smithi120.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 193 kB/s | 337 kB 00:01 2024-03-25T18:11:22.295 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:22.296 INFO:teuthology.orchestra.run.smithi120.stdout:Total 193 kB/s | 337 kB 00:01 2024-03-25T18:11:22.296 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:11:22.300 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:11:22.300 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:11:22.340 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:11:22.340 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:11:22.482 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:11:22.586 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:22.955 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:23.230 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:23.231 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:23.231 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:11:23.231 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:23.231 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:23.231 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:11:23.487 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:11:24.045 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:42:36 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:11:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:11:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:24.193 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:11:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:24.193 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:11:24.193 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:11:24.194 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:11:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:11:24.194 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:24.194 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 509 k 2024-03-25T18:11:24.195 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.5 M 2024-03-25T18:11:24.195 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:11:25.764 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 324 kB/s | 509 kB 00:01 2024-03-25T18:11:25.765 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:25.765 INFO:teuthology.orchestra.run.smithi120.stdout:Total 324 kB/s | 509 kB 00:01 2024-03-25T18:11:25.766 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:11:25.769 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:11:25.769 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:11:25.809 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:11:25.809 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:11:25.968 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:11:26.072 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:11:26.478 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:11:26.747 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:11:26.747 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:26.747 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:11:26.747 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:26.748 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:26.748 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:11:26.957 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:11:27.538 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:42:39 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:27.687 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:11:27.688 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:11:27.688 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:11:27.688 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:11:27.688 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 52 M 2024-03-25T18:11:27.688 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 298 M 2024-03-25T18:11:27.688 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:12:41.142 INFO:teuthology.orchestra.run.smithi042.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 299 kB/s | 44 MB 02:31 2024-03-25T18:12:41.143 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:41.143 INFO:teuthology.orchestra.run.smithi042.stdout:Total 299 kB/s | 44 MB 02:31 2024-03-25T18:12:41.143 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:12:41.146 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:12:41.147 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:12:41.459 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:12:41.459 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:12:46.812 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:12:46.918 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:12:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:12:49.120 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:12:49.120 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:12:49.120 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:12:49.120 INFO:teuthology.orchestra.run.smithi042.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:49.120 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:12:49.120 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:12:49.308 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:12:49.883 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:44:02 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:12:50.041 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 52 M 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 308 M 2024-03-25T18:12:50.042 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:13:52.307 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 365 kB/s | 52 MB 02:24 2024-03-25T18:13:52.307 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:52.308 INFO:teuthology.orchestra.run.smithi120.stdout:Total 365 kB/s | 52 MB 02:24 2024-03-25T18:13:52.308 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:13:52.310 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:13:52.310 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:13:52.644 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:13:52.644 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:13:58.538 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:13:58.788 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:14:00.998 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:14:01.289 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:14:01.289 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:14:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:14:01.289 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:01.289 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:14:01.289 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:14:01.536 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:14:02.099 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 1:45:14 ago on Mon 25 Mar 2024 04:28:48 PM UTC. 2024-03-25T18:14:02.246 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-25T18:14:02.247 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-25T18:14:02.248 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2024-03-25T18:14:02.248 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:14:02.248 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.3 M 2024-03-25T18:14:02.248 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 5.5 M 2024-03-25T18:14:02.248 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-25T18:14:06.683 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 289 kB/s | 1.3 MB 00:04 2024-03-25T18:14:06.683 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:14:06.684 INFO:teuthology.orchestra.run.smithi120.stdout:Total 289 kB/s | 1.3 MB 00:04 2024-03-25T18:14:06.684 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-25T18:14:06.688 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-25T18:14:06.688 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-25T18:14:06.732 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-25T18:14:06.732 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-25T18:14:06.973 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-25T18:14:07.081 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:14:07.530 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:14:07.830 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:14:07.830 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:14:07.830 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-25T18:14:07.830 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:07.830 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:14:07.830 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-25T18:14:08.053 DEBUG:teuthology.parallel:result is None 2024-03-25T18:15:42.796 INFO:teuthology.orchestra.run.smithi042.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 307 kB/s | 52 MB 02:52 2024-03-25T18:15:42.797 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:42.797 INFO:teuthology.orchestra.run.smithi042.stdout:Total 307 kB/s | 52 MB 02:52 2024-03-25T18:15:42.797 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:15:42.800 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:15:42.800 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:15:43.159 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:15:43.159 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:15:49.502 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:15:49.605 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:51.813 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:52.087 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:52.087 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:52.087 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:15:52.087 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:52.087 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:52.087 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:15:52.273 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:15:52.842 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:47:05 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:15:52.997 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:15:52.998 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:15:52.999 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:53.000 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 615 k 2024-03-25T18:15:53.000 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 2.1 M 2024-03-25T18:15:53.000 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:15:54.556 INFO:teuthology.orchestra.run.smithi042.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 395 kB/s | 615 kB 00:01 2024-03-25T18:15:54.557 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:54.558 INFO:teuthology.orchestra.run.smithi042.stdout:Total 394 kB/s | 615 kB 00:01 2024-03-25T18:15:54.558 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:15:54.562 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:15:54.562 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:15:54.606 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:15:54.606 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:15:54.782 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:15:54.883 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:55.337 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:55.602 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:55.602 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:55.602 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:15:55.602 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:55.602 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:55.602 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:15:55.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:15:56.349 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:47:09 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:15:56.505 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:15:56.506 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:15:56.506 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 771 k 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 2.7 M 2024-03-25T18:15:56.507 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:15:58.318 INFO:teuthology.orchestra.run.smithi042.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 426 kB/s | 771 kB 00:01 2024-03-25T18:15:58.319 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:58.319 INFO:teuthology.orchestra.run.smithi042.stdout:Total 425 kB/s | 771 kB 00:01 2024-03-25T18:15:58.319 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:15:58.323 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:15:58.323 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:15:58.366 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:15:58.367 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:15:58.554 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:15:58.645 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:59.034 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:59.303 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:59.304 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:59.304 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:15:59.304 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:59.304 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:15:59.304 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:15:59.498 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:16:00.071 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:47:13 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:16:00.228 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:16:00.229 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:00.230 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:16:00.230 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:00.230 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 1.1 M 2024-03-25T18:16:00.230 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 4.3 M 2024-03-25T18:16:00.230 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:16:06.270 INFO:teuthology.orchestra.run.smithi042.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 191 kB/s | 1.1 MB 00:06 2024-03-25T18:16:06.272 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:06.272 INFO:teuthology.orchestra.run.smithi042.stdout:Total 190 kB/s | 1.1 MB 00:06 2024-03-25T18:16:06.272 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:16:06.276 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:16:06.276 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:16:06.320 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:16:06.321 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:16:06.553 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:16:06.655 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:07.070 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:07.333 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:07.334 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:07.334 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:16:07.334 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:07.334 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:07.334 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:16:07.695 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:16:08.272 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:47:21 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:16:08.430 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:16:08.430 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:08.430 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:16:08.430 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:08.430 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:16:08.431 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:16:08.431 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:08.431 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:16:08.431 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:08.431 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:16:08.431 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:08.431 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 337 k 2024-03-25T18:16:08.432 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 1.0 M 2024-03-25T18:16:08.432 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:16:09.356 INFO:teuthology.orchestra.run.smithi042.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 365 kB/s | 337 kB 00:00 2024-03-25T18:16:09.357 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:09.357 INFO:teuthology.orchestra.run.smithi042.stdout:Total 364 kB/s | 337 kB 00:00 2024-03-25T18:16:09.357 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:16:09.361 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:16:09.361 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:16:09.400 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:16:09.400 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:16:09.549 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:16:09.667 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:10.044 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:10.301 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:10.301 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:10.301 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:16:10.301 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:10.301 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:10.301 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:16:10.518 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:16:11.094 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:47:24 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:16:11.249 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:16:11.249 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 509 k 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 2.5 M 2024-03-25T18:16:11.250 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:16:12.635 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 368 kB/s | 509 kB 00:01 2024-03-25T18:16:12.635 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:12.636 INFO:teuthology.orchestra.run.smithi042.stdout:Total 367 kB/s | 509 kB 00:01 2024-03-25T18:16:12.636 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:16:12.640 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:16:12.640 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:16:12.681 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:16:12.682 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:16:12.850 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:16:12.970 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:13.333 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:13.599 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:13.599 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:13.599 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:16:13.600 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:13.600 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:13.600 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:16:13.808 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:16:14.384 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:47:27 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:16:14.540 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:16:14.541 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:16:14.542 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 52 M 2024-03-25T18:16:14.542 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 298 M 2024-03-25T18:16:14.542 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:18:27.043 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 398 kB/s | 52 MB 02:12 2024-03-25T18:18:27.043 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:27.043 INFO:teuthology.orchestra.run.smithi042.stdout:Total 398 kB/s | 52 MB 02:12 2024-03-25T18:18:27.044 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:18:27.046 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:18:27.046 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:18:27.396 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:18:27.396 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:18:33.660 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:18:33.753 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:35.827 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:36.103 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:36.104 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:36.104 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:18:36.104 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:36.104 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:36.104 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:18:36.312 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:18:36.891 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 1:49:49 ago on Mon 25 Mar 2024 04:28:47 PM UTC. 2024-03-25T18:18:37.045 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-25T18:18:37.046 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:18:37.046 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-25T18:18:37.046 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 1.3 M 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 5.5 M 2024-03-25T18:18:37.047 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-25T18:18:41.343 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 298 kB/s | 1.3 MB 00:04 2024-03-25T18:18:41.344 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:41.345 INFO:teuthology.orchestra.run.smithi042.stdout:Total 298 kB/s | 1.3 MB 00:04 2024-03-25T18:18:41.345 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-25T18:18:41.349 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-25T18:18:41.349 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-25T18:18:41.398 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-25T18:18:41.399 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-25T18:18:41.653 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-25T18:18:41.753 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:42.219 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:42.481 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:42.481 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:42.482 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-25T18:18:42.482 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:42.482 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:42.482 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-25T18:18:42.693 DEBUG:teuthology.parallel:result is None 2024-03-25T18:18:42.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:18:42.834 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:18:42.854 INFO:teuthology.orchestra.run.smithi042.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:42.855 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:42.855 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:18:42.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:18:43.070 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:18:43.094 INFO:teuthology.orchestra.run.smithi120.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:43.094 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:43.094 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:18:43.095 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:18:43.095 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:43.096 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:18:43.123 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:18:43.123 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:18:43.165 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:18:43.165 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:43.165 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:18:43.192 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:18:43.258 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:18:43.258 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:18:43.289 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:18:43.359 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:18:43.360 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:43.360 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:18:43.386 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:18:43.453 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:18:43.453 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:18:43.482 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:18:43.550 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:18:43.550 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:43.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:18:43.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:18:43.643 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:18:43.643 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:18:43.671 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:18:43.739 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:18:43.820 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:18:43.820 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 777 /var/log/ceph 2024-03-25T18:18:43.823 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /var/log/ceph 2024-03-25T18:18:43.850 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:18:43.850 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:18:43.890 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:18:43.916 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:18:43.916 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:18:43.956 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:18:44.010 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:18:44.011 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:18:44.011 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621272', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi042.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8BW/raJ7hs8HkgBcrqosAu5QNhsG8VKh8OtuWAIjVOj/xngIyLwFiFziQPbI2DGtRp/gvAIt4u1l82IIosaco=', 'smithi120.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9c2Q9gPXMZsTiZJDGLRFyhiZQQwT0ovlYf6cy2hhSfX+o3b6NEreqMXCwN6ywjkx7gSe/R9PjyQUF8U/yPY4o='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:18:44.011 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:18:44.022 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:18:44.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:18:44.076 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:18:44.105 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:44.105 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:18:44.154 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:18:44.154 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-25 16:35:06.102624048 +0000 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-25 16:27:53.552665124 +0000 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-25 16:27:53.552665124 +0000 2024-03-25T18:18:44.209 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-25 16:27:53.552665124 +0000 2024-03-25T18:18:44.210 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:18:44.273 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-25T18:18:44.273 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-25T18:18:44.273 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000100005 s, 5.1 MB/s 2024-03-25T18:18:44.275 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:18:44.330 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-25 16:35:06.103624040 +0000 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-25 16:27:53.903662466 +0000 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-25 16:27:53.903662466 +0000 2024-03-25T18:18:44.384 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-25 16:27:53.903662466 +0000 2024-03-25T18:18:44.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:18:44.448 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-25T18:18:44.448 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-25T18:18:44.448 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 9.5308e-05 s, 5.4 MB/s 2024-03-25T18:18:44.449 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:18:44.505 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 2024-03-25T18:18:44.560 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:18:44.560 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:44.560 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:18:44.560 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:44.560 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:44.560 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-25 16:35:06.103624040 +0000 2024-03-25T18:18:44.560 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-25 16:27:54.224660036 +0000 2024-03-25T18:18:44.561 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-25 16:27:54.224660036 +0000 2024-03-25T18:18:44.561 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-25 16:27:54.224660036 +0000 2024-03-25T18:18:44.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:18:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-25T18:18:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-25T18:18:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 9.3789e-05 s, 5.5 MB/s 2024-03-25T18:18:44.626 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:18:44.682 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 2024-03-25T18:18:44.736 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:18:44.736 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:44.736 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:18:44.736 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:44.737 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:44.737 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-25 16:35:06.103624040 +0000 2024-03-25T18:18:44.737 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-25 16:27:54.565657454 +0000 2024-03-25T18:18:44.737 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-25 16:27:54.565657454 +0000 2024-03-25T18:18:44.737 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-25 16:27:54.565657454 +0000 2024-03-25T18:18:44.737 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:18:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-25T18:18:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-25T18:18:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 8.1001e-05 s, 6.3 MB/s 2024-03-25T18:18:44.803 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:18:44.857 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:44.857 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:18:44.858 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:18:44.874 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:18:44.874 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-25 16:35:41.692053810 +0000 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-25 16:27:59.023393244 +0000 2024-03-25T18:18:44.930 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-25 16:27:59.023393244 +0000 2024-03-25T18:18:44.931 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-25 16:27:59.023393244 +0000 2024-03-25T18:18:44.931 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:18:44.997 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-25T18:18:44.997 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-25T18:18:44.997 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000268204 s, 1.9 MB/s 2024-03-25T18:18:44.998 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:18:45.055 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-03-25T18:18:45.109 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:18:45.109 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:45.109 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:18:45.109 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:45.109 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:45.110 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-25 16:35:41.692053810 +0000 2024-03-25T18:18:45.110 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-25 16:27:59.364385417 +0000 2024-03-25T18:18:45.110 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-25 16:27:59.364385417 +0000 2024-03-25T18:18:45.110 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-25 16:27:59.364385417 +0000 2024-03-25T18:18:45.110 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:18:45.175 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-25T18:18:45.175 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-25T18:18:45.175 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000234084 s, 2.2 MB/s 2024-03-25T18:18:45.176 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:18:45.231 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-25 16:35:41.692053810 +0000 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-25 16:27:59.695377818 +0000 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-25 16:27:59.695377818 +0000 2024-03-25T18:18:45.286 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-25 16:27:59.695377818 +0000 2024-03-25T18:18:45.287 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:18:45.350 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-25T18:18:45.350 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-25T18:18:45.350 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000180755 s, 2.8 MB/s 2024-03-25T18:18:45.351 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:18:45.407 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-25 16:35:41.693053786 +0000 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-25 16:28:00.055369554 +0000 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-25 16:28:00.055369554 +0000 2024-03-25T18:18:45.462 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-25 16:28:00.055369554 +0000 2024-03-25T18:18:45.462 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:18:45.526 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-25T18:18:45.527 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-25T18:18:45.527 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000230486 s, 2.2 MB/s 2024-03-25T18:18:45.528 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:18:45.583 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:45.584 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:18:45.584 INFO:tasks.ceph:Generating config... 2024-03-25T18:18:45.585 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:18:45.585 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:18:45.585 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:18:45.585 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:18:45.585 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:18:45.585 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:18:45.585 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:18:45.585 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:18:45.585 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:18:45.585 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:18:45.585 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:18:45.585 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:18:45.585 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:18:45.586 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:18:45.586 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:18:45.586 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:18:45.586 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:18:45.586 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:18:45.586 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:18:45.586 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:18:45.587 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:18:45.588 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:18:45.588 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:18:45.588 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:18:45.639 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:18:45.642 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:18:45.688 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:18:45.715 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.42:6789]'), ('mon.c', '[v1:172.21.15.42:6790]'), ('mon.b', '[v1:172.21.15.120:6789]')] 2024-03-25T18:18:45.715 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.42:6789],[v1:172.21.15.42:6790],[v1:172.21.15.120:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:18:45.716 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:45.716 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:18:45.770 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.42:6789]' --addv c '[v1:172.21.15.42:6790]' --addv b '[v1:172.21.15.120:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:45.855 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:45.855 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: generated fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:18:45.855 INFO:teuthology.orchestra.run.smithi042.stdout:setting min_mon_release = quincy 2024-03-25T18:18:45.855 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 0 2024-03-25T18:18:45.855 INFO:teuthology.orchestra.run.smithi042.stdout:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:18:45.855 INFO:teuthology.orchestra.run.smithi042.stdout:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:18:45.855 INFO:teuthology.orchestra.run.smithi042.stdout:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:18:45.856 INFO:teuthology.orchestra.run.smithi042.stdout:min_mon_release 17 (quincy) 2024-03-25T18:18:45.856 INFO:teuthology.orchestra.run.smithi042.stdout:election_strategy: 3 2024-03-25T18:18:45.856 INFO:teuthology.orchestra.run.smithi042.stdout:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:18:45.856 INFO:teuthology.orchestra.run.smithi042.stdout:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:18:45.856 INFO:teuthology.orchestra.run.smithi042.stdout:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:18:45.856 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:18:45.857 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:18:45.871 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 402856df-369d-4d95-b0d1-b534080fb192... 2024-03-25T18:18:45.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:18:45.914 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:18:45.961 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2024-03-25T18:18:45.961 INFO:teuthology.orchestra.run.smithi042.stdout: chdir = "" 2024-03-25T18:18:45.961 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:18:45.961 INFO:teuthology.orchestra.run.smithi042.stdout: auth supported = cephx 2024-03-25T18:18:45.961 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.961 INFO:teuthology.orchestra.run.smithi042.stdout: filestore xattr use omap = true 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: mon clock drift allowed = 1.000 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: osd crush chooseleaf type = 0 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: auth debug = true 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on old message = true 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on bug = true 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: mon pg warn max object skew = 0 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:18:45.962 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default size = 2 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow primary affinity = true 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow pg remap = true 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on no sortbitwise = false 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on too few osds = false 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: mon_allow_pool_size_one = true 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.963 INFO:teuthology.orchestra.run.smithi042.stdout: osd default data pool replay window = 5 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: mon allow pool delete = true 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: mon cluster log file level = debug 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: debug asserts on shutdown = true 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: mon health detail to clog = false 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: mon host = "[v1:172.21.15.42:6789],[v1:172.21.15.42:6790],[v1:172.21.15.120:6789]" 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: mon client directed command retry = 5 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: mon election default strategy = 3 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: ms bind msgr2 = False 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: ms inject socket failures = 5000 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: ms type = async 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: osd heartbeat grace = 80 2024-03-25T18:18:45.964 INFO:teuthology.orchestra.run.smithi042.stdout: fsid = 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout:[osd] 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd journal size = 100 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub load threshold = 5.0 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub max interval = 600 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd recover clone overlap = true 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd recovery max chunk = 1048576 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug shutdown = true 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug op order = true 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify stray on activate = true 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug trim objects = true 2024-03-25T18:18:45.965 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: osd open classes on start = true 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug pg log writeout = true 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: osd map max advance = 10 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: journal zero on create = true 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: filestore apply finisher threads = 3 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: bdev debug aio = true 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug misdirected ops = true 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: bdev async discard = True 2024-03-25T18:18:45.966 INFO:teuthology.orchestra.run.smithi042.stdout: bdev enable discard = True 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore allocator = stupid 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore block size = 96636764160 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore fsck on mount = True 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore zero block detection = True 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluefs = 1 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluestore = 1 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: debug monc = 20 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: debug osd = 20 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: debug rocksdb = 10 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd full ratio = 0.9 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: osd class default list = * 2024-03-25T18:18:45.967 INFO:teuthology.orchestra.run.smithi042.stdout: osd class load list = * 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd client watch timeout = 120 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify cached snaps = True 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify missing on start = True 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd fast shutdown = False 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd max markdown count = 1000 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd mclock override recovery settings = True 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd objectstore = bluestore 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd op queue = debug_random 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd op queue cut off = debug_random 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: osd shutdown pgref assert = True 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr] 2024-03-25T18:18:45.968 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: debug mgr = 20 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: mgr/telemetry/nag = false 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout:[mon] 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: debug paxos = 20 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: mon data avail warn = 5 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: mon mgr mkfs grace = 240 2024-03-25T18:18:45.969 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd reporter subtree level = osd 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd prime pg temp = true 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: # 1m isn't quite enough 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.970 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: mon min osdmap epochs = 50 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd crush smoke test = False 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: mon osdmap full prune interval = 2 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: mon osdmap full prune min = 15 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: mon scrub interval = 300 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on pool no app = False 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: paxos service trim min = 10 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout:[client] 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: rgw cache enabled = true 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable ops log = true 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable usage log = true 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:18:45.971 INFO:teuthology.orchestra.run.smithi042.stdout: debug monc = 20 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi042.stdout: debug objecter = 20 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi042.stdout: debug rados = 20 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.a] 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.c] 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.b] 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-03-25T18:18:45.972 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.973 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:18:45.974 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "[v1:172.21.15.42:6789],[v1:172.21.15.42:6790],[v1:172.21.15.120:6789]" 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: mon client directed command retry = 5 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: mon election default strategy = 3 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: ms bind msgr2 = False 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: ms inject socket failures = 5000 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: ms type = async 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: osd heartbeat grace = 80 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:18:45.975 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug trim objects = true 2024-03-25T18:18:45.976 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: bdev async discard = True 2024-03-25T18:18:45.977 INFO:teuthology.orchestra.run.smithi120.stdout: bdev enable discard = True 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore allocator = stupid 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore zero block detection = True 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 1 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 1 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: debug monc = 20 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 10 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-03-25T18:18:45.978 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd class default list = * 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd class load list = * 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd client watch timeout = 120 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify cached snaps = True 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify missing on start = True 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd fast shutdown = False 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd max markdown count = 1000 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock override recovery settings = True 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd op queue = debug_random 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd op queue cut off = debug_random 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: osd shutdown pgref assert = True 2024-03-25T18:18:45.979 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-25T18:18:45.980 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-03-25T18:18:45.981 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon min osdmap epochs = 50 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd crush smoke test = False 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon osdmap full prune interval = 2 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon osdmap full prune min = 15 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon scrub interval = 300 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on pool no app = False 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: paxos service trim min = 10 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-03-25T18:18:45.982 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout: debug monc = 20 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout: debug objecter = 20 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout: debug rados = 20 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-03-25T18:18:45.983 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-03-25T18:18:45.984 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:18:45.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:18:46.038 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:18:46.038 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:46.039 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:18:46.053 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:46.053 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:18:46.107 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:18:46.107 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:46.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:18:46.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:18:46.185 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:18:46.186 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:46.240 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:18:46.240 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:18:46.240 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:18:46.240 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:18:46.280 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:18:46.280 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:46.335 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:18:46.335 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:18:46.335 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:18:46.390 INFO:teuthology.orchestra.run.smithi042.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:18:46.392 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:18:46.455 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:18:46.457 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:18:46.457 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:18:46.457 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:18:46.501 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:18:46.514 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:18:46.556 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:18:46.571 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:18:46.571 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:18:46.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:46.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:46.597 INFO:tasks.ceph:role: osd.0 2024-03-25T18:18:46.598 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:18:46.598 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:46.664 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:46.700 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-03-25T18:18:46.701 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-03-25T18:18:46.701 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:46.777 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:46.848 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:46.916 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:46.916 INFO:tasks.ceph:role: osd.1 2024-03-25T18:18:46.916 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:18:46.916 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:46.982 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:47.014 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-03-25T18:18:47.016 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-03-25T18:18:47.016 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:47.087 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:47.156 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:47.224 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:47.224 INFO:tasks.ceph:role: osd.2 2024-03-25T18:18:47.224 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:18:47.224 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:18:47.289 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:47.289 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:47.289 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:47.289 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:47.290 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:47.290 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:47.290 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:47.290 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:47.290 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:47.290 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:47.321 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-03-25T18:18:47.323 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-03-25T18:18:47.323 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:47.398 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:47.469 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:47.535 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:47.535 INFO:tasks.ceph:role: osd.3 2024-03-25T18:18:47.535 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:18:47.535 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:18:47.601 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:47.601 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:47.601 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:47.601 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:47.601 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:47.602 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:47.602 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:47.602 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:47.602 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:47.602 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:47.634 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-03-25T18:18:47.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-03-25T18:18:47.636 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:47.711 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:47.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:47.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:47.881+0000 7f9cd400d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:18:47.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:47.881+0000 7f9cd400d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:18:47.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:47.881+0000 7f9cd400d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:18:47.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:47.881+0000 7f9cd400d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:18:47.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:47.881+0000 7f9cd400d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:18:47.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:47.882+0000 7f9cd400d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:18:50.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:50.760 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:50.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:50.849+0000 7fc758811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:18:50.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:50.849+0000 7fc758811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:18:50.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:50.849+0000 7fc758811640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:18:50.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:50.849+0000 7fc758811640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:18:50.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:50.849+0000 7fc758811640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:18:50.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:50.849+0000 7fc758811640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:18:53.698 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:53.729 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:53.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:53.817+0000 7f9aae6f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:18:53.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:53.817+0000 7f9aae6f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:18:53.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:53.817+0000 7f9aae6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:18:53.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:53.817+0000 7f9aae6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:18:53.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:53.817+0000 7f9aae6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:18:53.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:53.818+0000 7f9aae6f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:18:56.653 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:56.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:56.774+0000 7fbca180f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:18:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:56.774+0000 7fbca180f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:18:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:56.774+0000 7fbca180f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:18:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:56.774+0000 7fbca180f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:18:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:56.774+0000 7fbca180f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:18:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:18:56.775+0000 7fbca180f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:18:59.616 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:59.647 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:18:59.678 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:59.679 INFO:tasks.ceph:role: osd.4 2024-03-25T18:18:59.679 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:18:59.679 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:59.757 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:59.758 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:19:00.270 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-25T18:19:00.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-25T18:19:00.273 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:19:00.423 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:19:00.460 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:19:00.529 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:19:00.599 INFO:tasks.ceph:role: osd.5 2024-03-25T18:19:00.599 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:19:00.599 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:19:00.635 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:19:00.635 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:19:00.635 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:19:00.636 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:19:00.636 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:19:00.636 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:19:00.636 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:19:00.636 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:19:00.636 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:19:00.636 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:19:01.085 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-25T18:19:01.088 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-25T18:19:01.088 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:19:01.183 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:19:01.217 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:19:01.285 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:19:01.285 INFO:tasks.ceph:role: osd.6 2024-03-25T18:19:01.285 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:19:01.285 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:19:01.361 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:19:01.361 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:19:01.361 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:19:01.361 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:19:01.361 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:19:01.362 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:19:01.362 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:19:01.362 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:19:01.362 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:19:01.362 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:19:01.859 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-25T18:19:01.861 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-25T18:19:01.861 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:19:01.991 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:19:02.026 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:19:02.094 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:19:02.094 INFO:tasks.ceph:role: osd.7 2024-03-25T18:19:02.094 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:19:02.094 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:19:02.170 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:19:02.676 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-25T18:19:02.678 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-25T18:19:02.678 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:19:02.825 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:19:02.863 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:19:02.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:02.966+0000 7fdc209bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:19:02.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:02.966+0000 7fdc209bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:19:02.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:02.966+0000 7fdc209bd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:19:02.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:02.966+0000 7fdc209bd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:19:02.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:02.966+0000 7fdc209bd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:19:02.969 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:02.967+0000 7fdc209bd640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:19:05.822 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:19:05.853 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:19:05.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:05.943+0000 7f5e7c1bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:19:05.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:05.943+0000 7f5e7c1bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:19:05.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:05.943+0000 7f5e7c1bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:19:05.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:05.943+0000 7f5e7c1bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:19:05.945 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:05.943+0000 7f5e7c1bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:19:05.946 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:05.944+0000 7f5e7c1bb640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:19:08.790 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:19:08.821 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:19:08.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:08.913+0000 7f37f6617640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:19:08.914 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:08.913+0000 7f37f6617640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:19:08.915 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:08.913+0000 7f37f6617640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:19:08.915 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:08.913+0000 7f37f6617640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:19:08.915 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:08.913+0000 7f37f6617640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:19:08.915 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:08.914+0000 7f37f6617640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:19:11.761 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:19:11.792 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:19:11.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:11.882+0000 7f7b7931d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:19:11.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:11.882+0000 7f7b7931d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:19:11.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:11.882+0000 7f7b7931d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:19:11.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:11.882+0000 7f7b7931d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:19:11.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:11.882+0000 7f7b7931d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:19:11.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:11.882+0000 7f7b7931d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:19:14.726 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:19:14.756 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:19:14.757 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:14.757 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:19:14.788 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:14.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:19:14.859 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:14.859 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:19:14.929 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:14.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:19:14.998 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:14.998 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:19:15.067 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:15.067 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:19:15.099 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:15.099 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:19:15.168 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:15.168 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:19:15.237 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:15.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:19:15.304 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:15.304 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:19:15.372 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:15.372 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:19:15.389 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:15.389 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:19:15.427 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:19:15.427 DEBUG:teuthology.orchestra.run.smithi042:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:19:15.431 DEBUG:teuthology.orchestra.run.smithi120:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:19:15.470 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr.y] 2024-03-25T18:19:15.470 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAGwAFmG7REFxAAWXMbcSGLIl9GcM57l12wug== 2024-03-25T18:19:15.470 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.0] 2024-03-25T18:19:15.470 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAHwAFmAymdNBAAcKSDTDV8IvcIIoy48e8fKA== 2024-03-25T18:19:15.470 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.1] 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAKwAFm03euMhAAAAlbPR8tfd/vflteIc06Ww== 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.2] 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQANwAFmkmXNMBAA2M2Qj7w+iCREfr3SMrg6Eg== 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.3] 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAQwAFmKOI6LhAAgTiOjILb5kNt3QiYuENQnA== 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr.x] 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAGwAFmE6clGxAAlL3qDGVLqaUTiRjJ/fxCfA== 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.4] 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAWwAFmBTa4ORAAt2/6SFIO9a33NvqqsPaxzg== 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.5] 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAZwAFm/UVWOBAAM0GYHU/uE8qNQ0ddlZgZ3Q== 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.6] 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAcwAFm3sN/NhAAOKCzufe/6ByX5zwRoJWfdQ== 2024-03-25T18:19:15.471 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.7] 2024-03-25T18:19:15.472 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAfwAFmtEOhNBAAw6IpuIXCBrrJ+9xzbqfYLA== 2024-03-25T18:19:15.472 INFO:teuthology.orchestra.run.smithi042.stdout:[client.0] 2024-03-25T18:19:15.472 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAGwAFmwd7GHRAA2uc2ewHza6fsLTdxiSmppA== 2024-03-25T18:19:15.472 INFO:teuthology.orchestra.run.smithi042.stdout:[client.1] 2024-03-25T18:19:15.472 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAGwAFm9QUzIRAAfvFE8g95f5gCV1l2O3vGaw== 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.y] 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAGwAFmG7REFxAAWXMbcSGLIl9GcM57l12wug== 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAHwAFmAymdNBAAcKSDTDV8IvcIIoy48e8fKA== 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAKwAFm03euMhAAAAlbPR8tfd/vflteIc06Ww== 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQANwAFmkmXNMBAA2M2Qj7w+iCREfr3SMrg6Eg== 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.3] 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAQwAFmKOI6LhAAgTiOjILb5kNt3QiYuENQnA== 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAGwAFmE6clGxAAlL3qDGVLqaUTiRjJ/fxCfA== 2024-03-25T18:19:15.493 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.4] 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAWwAFmBTa4ORAAt2/6SFIO9a33NvqqsPaxzg== 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.5] 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAZwAFm/UVWOBAAM0GYHU/uE8qNQ0ddlZgZ3Q== 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.6] 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAcwAFm3sN/NhAAOKCzufe/6ByX5zwRoJWfdQ== 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.7] 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAfwAFmtEOhNBAAw6IpuIXCBrrJ+9xzbqfYLA== 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout:[client.0] 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAGwAFmwd7GHRAA2uc2ewHza6fsLTdxiSmppA== 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout:[client.1] 2024-03-25T18:19:15.494 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQAGwAFm9QUzIRAAfvFE8g95f5gCV1l2O3vGaw== 2024-03-25T18:19:15.495 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:19:15.515 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:19:15.587 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.589 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.643 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.697 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.750 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.801 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:19:15.803 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:19:15.856 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.858 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.909 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.911 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.964 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:15.966 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:16.018 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:16.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:19:16.072 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:19:16.074 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:19:16.128 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:19:16.131 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:19:16.186 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:19:16.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:19:16.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:19:16.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:19:16.335 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:19:16.401 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:19:16.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:19:16.523 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:19:16.550 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:19:16.646 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:19:16.672 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:19:16.674 DEBUG:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:19:16.727 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:19:16.727 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:19:16.727 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:19:16.728 DEBUG:teuthology.orchestra.run.smithi042:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:19:16.729 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:19:16.730 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:19:16.730 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:19:16.730 DEBUG:teuthology.orchestra.run.smithi042:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:19:16.732 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:19:16.732 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:19:16.732 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:19:16.732 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:19:16.770 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:19:16.770 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:19:16.770 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:19:16.770 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:19:16.772 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:19:16.773 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:19:16.773 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:19:16.775 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:19:16.775 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:19:16.775 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:19:16.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.875+0000 7f9e3a6ba640 10 monclient: get_monmap_and_config 2024-03-25T18:19:16.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.875+0000 7f9e3a6ba640 10 monclient: build_initial_monmap 2024-03-25T18:19:16.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.875+0000 7f9e3a6ba640 10 monclient: monmap: 2024-03-25T18:19:16.876 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-03-25T18:19:16.876 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:16.876 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:19:16.876768+0000 2024-03-25T18:19:16.877 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:19:16.876768+0000 2024-03-25T18:19:16.877 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:16.877 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:16.877 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:16.877 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:16.877 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:16.877 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:16.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.876+0000 7f9e3a6ba640 1 Processor -- start 2024-03-25T18:19:16.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.876+0000 7f9e3a6ba640 1 -- start start 2024-03-25T18:19:16.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.876+0000 7f9e3a6ba640 10 monclient: init 2024-03-25T18:19:16.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f9e340a9920 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9e340b45a0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f9e34146450 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3406b100 con 0x7f9e34146450 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e340b2940 con 0x7f9e340b45a0 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3413fec0 con 0x7f9e340a9920 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e3a6ba640 10 monclient(hunting): authenticate will time out at 7272.103649s 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e3414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e3413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:16.877+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e34145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:17.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:17.078+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e3413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:17.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:17.078+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e3414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:17.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:17.078+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e34145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:17.479+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e3413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:17.479+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e3414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:17.479+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e34145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:18.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:18.280+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e3413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:18.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:18.280+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e3414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:18.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:18.280+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e34145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:19.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient(hunting): tick 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e3413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e34145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e3414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f9e340a9920 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9e34146450 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9e340b45a0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3406b100 con 0x7f9e34146450 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e340b2940 con 0x7f9e340b45a0 2024-03-25T18:19:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3413fec0 con 0x7f9e340a9920 2024-03-25T18:19:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e31ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e337fe640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:19.877+0000 7f9e38c57640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e2000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:20.078+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:20.078+0000 7f9e337fe640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:20.078+0000 7f9e38c57640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e2000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:20.479+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:20.479+0000 7f9e337fe640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:20.479+0000 7f9e38c57640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e2000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:21.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:21.280+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:21.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:21.280+0000 7f9e337fe640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:21.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:21.280+0000 7f9e38c57640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e2000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:22.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient(hunting): tick 2024-03-25T18:19:22.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:22.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e2000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f9e340a9920 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9e340b45a0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9e34146450 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20010e20 con 0x7f9e34146450 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3406b100 con 0x7f9e340b45a0 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e340b2940 con 0x7f9e340a9920 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e31ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:22.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:22.878+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:23.079+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:23.079+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:23.079+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:23.480+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:23.480+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:23.480+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:23.752 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:00:06.873 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:24.062 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:00:07.182 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:24.209 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:00:07.378 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:24.217 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:00:07.386 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:24.281+0000 7f9e33fff640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6790/0 2024-03-25T18:19:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:24.281+0000 7f9e38c57640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-03-25T18:19:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:24.281+0000 7f9e337fe640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-25T18:19:24.535 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:00:07.705 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:24.544 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:00:07.713 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient(hunting): tick 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 1 -- >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9e34146450 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f9e340a9920 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9e340b45a0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3413fec0 con 0x7f9e34146450 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20010e20 con 0x7f9e340b45a0 2024-03-25T18:19:25.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3406b100 con 0x7f9e340a9920 2024-03-25T18:19:25.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.877+0000 7f9e31ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:25.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.928+0000 7f9e38c57640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 0x7f9e200105a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:38028/0 (socket says 172.21.15.42:38028) 2024-03-25T18:19:25.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:25.928+0000 7f9e38c57640 1 -- 172.21.15.42:0/994325742 learned_addr learned my addr 172.21.15.42:0/994325742 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:26.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:26.210+0000 7f9e337fe640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T18:19:26.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:26.210+0000 7f9e337fe640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:19:26.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:26.210+0000 7f9e337fe640 1 --1- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 0x7f9e2000fdf0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:19:26.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:26.242+0000 7f9e33fff640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:19:26.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:26.242+0000 7f9e33fff640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:19:26.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:26.242+0000 7f9e33fff640 1 --1- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 0x7f9e20008ea0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:19:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient(hunting): tick 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e340a9920 legacy=0x7f9e2000fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e200105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e34146450 legacy=0x7f9e20008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2000d6d0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9e200120a0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9e340b45a0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e340b2940 con 0x7f9e200120a0 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2000fdf0 con 0x7f9e340b45a0 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2000ff80 con 0x7f9e2000d6d0 2024-03-25T18:19:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:28.877+0000 7f9e31ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient(hunting): tick 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e2000d6d0 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.120:6789/0 conn(0x7f9e340b45a0 legacy=0x7f9e200105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e200120a0 legacy=0x7f9e200095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9e20005e00 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9e20008eb0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2000d6d0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200101d0 con 0x7f9e20008eb0 2024-03-25T18:19:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20009290 con 0x7f9e20005e00 2024-03-25T18:19:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200095d0 con 0x7f9e2000d6d0 2024-03-25T18:19:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:31.878+0000 7f9e31ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:34.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient(hunting): tick 2024-03-25T18:19:34.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:34.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e2000d6d0 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.120:6789/0 conn(0x7f9e20005e00 legacy=0x7f9e2000bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e20008eb0 legacy=0x7f9e200105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f9e200098f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9e20014490 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9e20011090 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20009760 con 0x7f9e20011090 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200105a0 con 0x7f9e20014490 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20010780 con 0x7f9e200098f0 2024-03-25T18:19:34.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:34.878+0000 7f9e31ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient(hunting): tick 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e200098f0 legacy=0x7f9e20006240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.120:6789/0 conn(0x7f9e20014490 legacy=0x7f9e20016880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e20011090 legacy=0x7f9e200056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9e20011470 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2000c690 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9e20011090 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200109b0 con 0x7f9e20011090 2024-03-25T18:19:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2000ca70 con 0x7f9e20011470 2024-03-25T18:19:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200056e0 con 0x7f9e2000c690 2024-03-25T18:19:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:37.878+0000 7f9e31ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:40.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.289+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444966074 0 0) 0x7f9e200109b0 con 0x7f9e20011090 2024-03-25T18:19:40.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.289+0000 7f9e32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.289+0000 7f9e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.289+0000 7f9e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.289+0000 7f9e32ffd640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:19:40.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.289+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e1c003520 con 0x7f9e20011090 2024-03-25T18:19:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.318+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439698554 0 0) 0x7f9e2000ca70 con 0x7f9e20011470 2024-03-25T18:19:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.318+0000 7f9e32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.318+0000 7f9e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.318+0000 7f9e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.318+0000 7f9e32ffd640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:19:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.318+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e200109b0 con 0x7f9e20011470 2024-03-25T18:19:40.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.350+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977636035 0 0) 0x7f9e200056e0 con 0x7f9e2000c690 2024-03-25T18:19:40.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.350+0000 7f9e32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.350+0000 7f9e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.350+0000 7f9e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.350+0000 7f9e32ffd640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:19:40.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.350+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2000ca70 con 0x7f9e2000c690 2024-03-25T18:19:40.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421090348 0 0) 0x7f9e1c003520 con 0x7f9e20011090 2024-03-25T18:19:40.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6790/0 conn(0x7f9e2000c690 legacy=0x7f9e2000bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.120:6789/0 conn(0x7f9e20011470 legacy=0x7f9e20011850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 1 monclient: found mon.noname-a 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e200058c0 con 0x7f9e20011090 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9e2c0042f0 con 0x7f9e20011090 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e3a6ba640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e3a6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e32ffd640 10 monclient: dump: 2024-03-25T18:19:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.357+0000 7f9e3a6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e2c0044c0 con 0x7f9e20011090 2024-03-25T18:19:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e32ffd640 1 -- 172.21.15.42:0/994325742 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9e2c0048d0 con 0x7f9e20011090 2024-03-25T18:19:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e32ffd640 10 monclient: dump: 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e3a6ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.388+0000 7f9e3a6ba640 4 set_mon_vals no callback set 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.389+0000 7f9e3a6ba640 10 monclient: shutdown 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.389+0000 7f9e3a6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.389+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/994325742 >> v1:172.21.15.42:6789/0 conn(0x7f9e20011090 legacy=0x7f9e20006240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.389+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/994325742 shutdown_connections 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.389+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/994325742 >> 172.21.15.42:0/994325742 conn(0x7f9e340b7f40 msgr2=0x7f9e340b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.389+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/994325742 shutdown_connections 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.389+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/994325742 wait complete. 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.390+0000 7f9e3a6ba640 10 monclient: build_initial_monmap 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.390+0000 7f9e3a6ba640 10 monclient: monmap: 2024-03-25T18:19:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.390+0000 7f9e3a6ba640 1 librados: starting msgr at 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.390+0000 7f9e3a6ba640 1 librados: starting objecter 2024-03-25T18:19:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.390+0000 7f9e3a6ba640 1 Processor -- start 2024-03-25T18:19:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 1 -- start start 2024-03-25T18:19:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 1 librados: setting wanted keys 2024-03-25T18:19:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 1 librados: calling monclient init 2024-03-25T18:19:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient: init 2024-03-25T18:19:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9e2000c690 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f9e20011090 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f9e20011470 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e341412d0 con 0x7f9e20011090 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e34140d80 con 0x7f9e20011470 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e34141e70 con 0x7f9e2000c690 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.391+0000 7f9e3a6ba640 10 monclient(hunting): authenticate will time out at 7295.618161s 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.392+0000 7f9e33fff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f9e2000c690 0x7f9e34141760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:51294/0 (socket says 172.21.15.42:51294) 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.392+0000 7f9e33fff640 1 -- 172.21.15.42:0/1777807855 learned_addr learned my addr 172.21.15.42:0/1777807855 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.392+0000 7f9e30ff9640 10 client.?.objecter ms_handle_connect 0x7f9e2000c690 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.392+0000 7f9e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.392+0000 7f9e30ff9640 10 client.?.objecter ms_handle_connect 0x7f9e20011470 2024-03-25T18:19:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.392+0000 7f9e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:40.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.446+0000 7f9e30ff9640 10 client.?.objecter ms_handle_connect 0x7f9e20011090 2024-03-25T18:19:40.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.446+0000 7f9e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.453+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700110658 0 0) 0x7f9e341412d0 con 0x7f9e20011090 2024-03-25T18:19:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.453+0000 7f9e30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.453+0000 7f9e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.453+0000 7f9e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.453+0000 7f9e30ff9640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:19:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.453+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2000d3c0 con 0x7f9e20011090 2024-03-25T18:19:40.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.472+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558894054 0 0) 0x7f9e34140d80 con 0x7f9e20011470 2024-03-25T18:19:40.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.472+0000 7f9e30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.472+0000 7f9e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.472+0000 7f9e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.472+0000 7f9e30ff9640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:19:40.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.472+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e341412d0 con 0x7f9e20011470 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.477+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893488873 0 0) 0x7f9e2000d3c0 con 0x7f9e20011090 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.477+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e34140d80 con 0x7f9e20011090 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.477+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9e280042e0 con 0x7f9e20011090 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.477+0000 7f9e30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.477+0000 7f9e30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.477+0000 7f9e30ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.479 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.479 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:40.479 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:40.479 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:40.479 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:40.479 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:40.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.477+0000 7f9e30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.481+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180475293 0 0) 0x7f9e34141e70 con 0x7f9e2000c690 2024-03-25T18:19:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.481+0000 7f9e30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.481+0000 7f9e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.481+0000 7f9e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.481+0000 7f9e30ff9640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:19:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.481+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2000d3c0 con 0x7f9e2000c690 2024-03-25T18:19:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 18003554 0 0) 0x7f9e34140d80 con 0x7f9e20011090 2024-03-25T18:19:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 >> v1:172.21.15.42:6790/0 conn(0x7f9e2000c690 legacy=0x7f9e34141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 >> v1:172.21.15.120:6789/0 conn(0x7f9e20011470 legacy=0x7f9e341e3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 1 monclient: found mon.a 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34142170 con 0x7f9e20011090 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 10 monclient: _renew_subs 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e341423c0 con 0x7f9e20011090 2024-03-25T18:19:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 10 monclient: _renew_subs 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e341dbc80 con 0x7f9e20011090 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.497+0000 7f9e3a6ba640 1 librados: init done 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.498+0000 7f9e3a6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.498+0000 7f9e3a6ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.498+0000 7f9e3a6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.498+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e341dbc80 con 0x7f9e20011090 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e280046f0 con 0x7f9e20011090 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9e28004a40 con 0x7f9e20011090 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 10 monclient: dump: 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:40.727 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9e28004ff0 con 0x7f9e20011090 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1237988987 0 0) 0x7f9e28006480 con 0x7f9e20011090 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 10 client.4107.objecter ms_dispatch 0x7f9e34000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:19:40.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:19:40.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:19:40.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e327fc640 4 set_mon_vals no callback set 2024-03-25T18:19:40.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:19:40.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9e280068a0 con 0x7f9e20011090 2024-03-25T18:19:40.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:40.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.725+0000 7f9e30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:40.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.815+0000 7f9e3a6ba640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:19:40.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.815+0000 7f9e3a6ba640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:19:40.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.815+0000 7f9e3a6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:40.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:40.815+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9e341dbc80 con 0x7f9e20011090 2024-03-25T18:19:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.050+0000 7f9e30ff9640 1 -- 172.21.15.42:0/1777807855 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f9e280055b0 con 0x7f9e20011090 2024-03-25T18:19:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.050+0000 7f9e30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:19:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.050+0000 7f9e30ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:19:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:adjusted tunables profile to default 2024-03-25T18:19:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 10 librados: watch_flush enter 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 10 librados: watch_flush exit 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 10 monclient: shutdown 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 >> v1:172.21.15.42:6789/0 conn(0x7f9e20011090 legacy=0x7f9e34136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 shutdown_connections 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 >> 172.21.15.42:0/1777807855 conn(0x7f9e340b7f40 msgr2=0x7f9e34074530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 shutdown_connections 2024-03-25T18:19:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 1 -- 172.21.15.42:0/1777807855 wait complete. 2024-03-25T18:19:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:41.051+0000 7f9e3a6ba640 1 librados: shutdown 2024-03-25T18:19:41.063 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:19:41.063 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:19:41.064 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:41.064 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:19:41.092 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:41.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:19:41.160 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:41.160 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:19:41.227 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-25T18:19:41.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:19:41.264 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:41.263+0000 7f8cc6e3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:19:41.291 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:41.292 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:19:41.319 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:41.319 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:19:41.348 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:41.347+0000 7f8cc6e3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:19:41.356 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:41.355+0000 7f31ee442040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:19:41.383 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:41.383 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:19:41.413 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:41.412+0000 7f31ee442040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:19:41.439 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-25T18:19:41.439 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:19:41.506 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 300de658-80a1-43d5-91bb-a6be671335e8 0 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.645+0000 7f40b4d32640 10 monclient: get_monmap_and_config 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.645+0000 7f40b4d32640 10 monclient: build_initial_monmap 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.645+0000 7f40b4d32640 10 monclient: monmap: 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:41.646501+0000 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:41.646501+0000 2024-03-25T18:19:41.646 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:41.647 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:41.647 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:41.647 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:41.647 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:41.647 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:41.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 1 Processor -- start 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 1 -- start start 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient: init 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient(hunting): picked mon.noname-a con 0x7f40b00b4700 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient(hunting): picked mon.noname-b con 0x7f40b0142a20 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient(hunting): picked mon.noname-c con 0x7f40b00a96b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00a9d60 con 0x7f40b00b4700 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b013ff10 con 0x7f40b00a96b0 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00aa1f0 con 0x7f40b0142a20 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:41.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.646+0000 7f40b4d32640 10 monclient(hunting): authenticate will time out at 7299.802863s 2024-03-25T18:19:41.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.647+0000 7f40aed9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f40b00a96b0 0x7f40b013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47362/0 (socket says 172.21.15.120:47362) 2024-03-25T18:19:41.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.647+0000 7f40aed9d640 1 -- 172.21.15.120:0/2092019528 learned_addr learned my addr 172.21.15.120:0/2092019528 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:41.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.649+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458644310 0 0) 0x7f40b013ff10 con 0x7f40b00a96b0 2024-03-25T18:19:41.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.649+0000 7f40add9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:41.650 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.649+0000 7f40add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:41.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.649+0000 7f40add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:41.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.649+0000 7f40add9b640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:19:41.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.649+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c003520 con 0x7f40b00a96b0 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.650+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755766221 0 0) 0x7f40b00aa1f0 con 0x7f40b0142a20 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.650+0000 7f40add9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.650+0000 7f40add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.650+0000 7f40add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.650+0000 7f40add9b640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.650+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b013ff10 con 0x7f40b0142a20 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.651+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138492550 0 0) 0x7f40b00a9d60 con 0x7f40b00b4700 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.651+0000 7f40add9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.651+0000 7f40add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.651+0000 7f40add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:41.652 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.651+0000 7f40add9b640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:19:41.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.651+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b00aa1f0 con 0x7f40b00b4700 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25409777 0 0) 0x7f40b00aa1f0 con 0x7f40b00b4700 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 >> v1:172.21.15.42:6790/0 conn(0x7f40b0142a20 legacy=0x7f40b013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 >> v1:172.21.15.120:6789/0 conn(0x7f40b00a96b0 legacy=0x7f40b013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 1 monclient: found mon.noname-a 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b00b4ae0 con 0x7f40b00b4700 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 10 monclient: _finish_auth 0 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:41.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f40a00015b0 con 0x7f40b00b4700 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40b4d32640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.655+0000 7f40b4d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.656+0000 7f40add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.656+0000 7f40add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.656+0000 7f40add9b640 10 monclient: dump: 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.658 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:41.659 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:41.659 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:41.659 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:41.659 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:41.659 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.656+0000 7f40b4d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:41.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a0001710 con 0x7f40b00b4700 2024-03-25T18:19:41.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:41.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40add9b640 1 -- 172.21.15.120:0/2092019528 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f40a0001b60 con 0x7f40b00b4700 2024-03-25T18:19:41.661 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40add9b640 10 monclient: dump: 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:41.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 4 set_mon_vals no callback set 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 10 monclient: shutdown 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 1 -- 172.21.15.120:0/2092019528 >> v1:172.21.15.42:6789/0 conn(0x7f40b00b4700 legacy=0x7f40b0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 1 -- 172.21.15.120:0/2092019528 shutdown_connections 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 1 -- 172.21.15.120:0/2092019528 >> 172.21.15.120:0/2092019528 conn(0x7f40b00668e0 msgr2=0x7f40b0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 1 -- 172.21.15.120:0/2092019528 shutdown_connections 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.658+0000 7f40b4d32640 1 -- 172.21.15.120:0/2092019528 wait complete. 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.659+0000 7f40b4d32640 10 monclient: build_initial_monmap 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.659+0000 7f40b4d32640 10 monclient: monmap: 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:41.663 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.659+0000 7f40b4d32640 1 librados: starting msgr at 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.659+0000 7f40b4d32640 1 librados: starting objecter 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.659+0000 7f40b4d32640 1 Processor -- start 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 1 -- start start 2024-03-25T18:19:41.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 1 librados: setting wanted keys 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 1 librados: calling monclient init 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient: init 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient(hunting): picked mon.noname-c con 0x7f40b00a96b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient(hunting): picked mon.noname-b con 0x7f40b00b4700 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient(hunting): picked mon.noname-a con 0x7f40b0142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b0138bd0 con 0x7f40b0142a20 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b01398a0 con 0x7f40b00b4700 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b0139b80 con 0x7f40b00a96b0 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:41.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.660+0000 7f40b4d32640 10 monclient(hunting): authenticate will time out at 7299.816422s 2024-03-25T18:19:41.666 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.661+0000 7f40aed9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f40b0142a20 0x7f40b01e26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.120:46858/0 (socket says 172.21.15.120:46858) 2024-03-25T18:19:41.666 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.661+0000 7f40aed9d640 1 -- 172.21.15.120:0/4159006065 learned_addr learned my addr 172.21.15.120:0/4159006065 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:41.666 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.661+0000 7f40a6ffd640 10 client.?.objecter ms_handle_connect 0x7f40b0142a20 2024-03-25T18:19:41.666 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.661+0000 7f40a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:41.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.664+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744277847 0 0) 0x7f40b0138bd0 con 0x7f40b0142a20 2024-03-25T18:19:41.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.664+0000 7f40a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:41.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.664+0000 7f40a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:41.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.664+0000 7f40a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:41.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.664+0000 7f40a6ffd640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:19:41.667 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.664+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4090004eb0 con 0x7f40b0142a20 2024-03-25T18:19:41.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.672+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2176069783 0 0) 0x7f4090004eb0 con 0x7f40b0142a20 2024-03-25T18:19:41.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.672+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40b0138bd0 con 0x7f40b0142a20 2024-03-25T18:19:41.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.672+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f40a8001b70 con 0x7f40b0142a20 2024-03-25T18:19:41.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.672+0000 7f40a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.672+0000 7f40a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:41.673 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.672+0000 7f40a6ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:41.673 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:41.674 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.672+0000 7f40a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:41.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2113411184 0 0) 0x7f40b0138bd0 con 0x7f40b0142a20 2024-03-25T18:19:41.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 >> v1:172.21.15.42:6790/0 conn(0x7f40b00a96b0 legacy=0x7f40b012ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 >> v1:172.21.15.120:6789/0 conn(0x7f40b00b4700 legacy=0x7f40b0130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.679 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 1 monclient: found mon.a 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0139e60 con 0x7f40b0142a20 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 10 monclient: _renew_subs 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b01385d0 con 0x7f40b0142a20 2024-03-25T18:19:41.680 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 10 monclient: _renew_subs 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b0137a70 con 0x7f40b0142a20 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.677+0000 7f40b4d32640 1 librados: init done 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.678+0000 7f40b4d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.678+0000 7f40b4d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.678+0000 7f40b4d32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:41.681 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.678+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b004f730 con 0x7f40b0142a20 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a8001cd0 con 0x7f40b0142a20 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f40a8004d80 con 0x7f40b0142a20 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40a6ffd640 10 monclient: dump: 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.796 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40ad59a640 4 set_mon_vals no callback set 2024-03-25T18:19:41.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.795+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f40a8005330 con 0x7f40b0142a20 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 678326765 0 0) 0x7f40a8006870 con 0x7f40b0142a20 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 10 client.4119.objecter ms_dispatch 0x7f40b0000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f40a8006c00 con 0x7f40b0142a20 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:41.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.796+0000 7f40a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:41.869 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.869+0000 7f40b4d32640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "300de658-80a1-43d5-91bb-a6be671335e8", "id": 0}] 2024-03-25T18:19:41.870 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.869+0000 7f40b4d32640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "300de658-80a1-43d5-91bb-a6be671335e8", "id": 0}] 2024-03-25T18:19:41.870 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.869+0000 7f40b4d32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:41.870 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.869+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd new", "uuid": "300de658-80a1-43d5-91bb-a6be671335e8", "id": 0} v 0) v1 -- 0x7f40b00b6a60 con 0x7f40b0142a20 2024-03-25T18:19:41.893 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:41.892+0000 7f8cc6e3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:19:41.949 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:41.948+0000 7f8cc6e3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:19:41.963 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:41.962+0000 7f31ee442040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:19:41.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.980+0000 7f40a6ffd640 1 -- 172.21.15.120:0/4159006065 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "300de658-80a1-43d5-91bb-a6be671335e8", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2437193101 0 732707515) 0x7f40a80058e0 con 0x7f40b0142a20 2024-03-25T18:19:41.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.980+0000 7f40a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "300de658-80a1-43d5-91bb-a6be671335e8", "id": 0}] 2024-03-25T18:19:41.981 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.980+0000 7f40a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:41.981 INFO:teuthology.orchestra.run.smithi120.stdout:0 2024-03-25T18:19:41.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 10 librados: watch_flush enter 2024-03-25T18:19:41.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 10 librados: watch_flush exit 2024-03-25T18:19:41.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:19:41.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:19:41.983 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 10 monclient: shutdown 2024-03-25T18:19:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 >> v1:172.21.15.42:6789/0 conn(0x7f40b0142a20 legacy=0x7f40b01e26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 shutdown_connections 2024-03-25T18:19:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 >> 172.21.15.120:0/4159006065 conn(0x7f40b00668e0 msgr2=0x7f40b00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.981+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 shutdown_connections 2024-03-25T18:19:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.982+0000 7f40b4d32640 1 -- 172.21.15.120:0/4159006065 wait complete. 2024-03-25T18:19:41.984 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:41.982+0000 7f40b4d32640 1 librados: shutdown 2024-03-25T18:19:41.996 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 76fae7c0-88cd-4858-898c-ee21e1b87701 1 2024-03-25T18:19:41.999 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:41.998+0000 7f8cc6e3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:19:42.020 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.020+0000 7f31ee442040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:19:42.055 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.054+0000 7f8cc6e3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:19:42.068 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.067+0000 7f31ee442040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:19:42.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.095+0000 7f95f412b640 10 monclient: get_monmap_and_config 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.095+0000 7f95f412b640 10 monclient: build_initial_monmap 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.095+0000 7f95f412b640 10 monclient: monmap: 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:42.096572+0000 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:42.096572+0000 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.097 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:42.098 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:42.098 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:42.098 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 1 Processor -- start 2024-03-25T18:19:42.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 1 -- start start 2024-03-25T18:19:42.098 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient: init 2024-03-25T18:19:42.099 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec0b4700 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec142a20 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec0a96b0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0a9d60 con 0x7f95ec0a96b0 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec13ff10 con 0x7f95ec142a20 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0aa1f0 con 0x7f95ec0b4700 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.096+0000 7f95f412b640 10 monclient(hunting): authenticate will time out at 7300.252874s 2024-03-25T18:19:42.100 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.097+0000 7f95f16c6640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f95ec142a20 0x7f95ec13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47386/0 (socket says 172.21.15.120:47386) 2024-03-25T18:19:42.101 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.097+0000 7f95f16c6640 1 -- 172.21.15.120:0/1953000924 learned_addr learned my addr 172.21.15.120:0/1953000924 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:42.101 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.099+0000 7f8cc6e3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.099+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753825925 0 0) 0x7f95ec13ff10 con 0x7f95ec142a20 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.099+0000 7f95f0ec5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.099+0000 7f95f0ec5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.099+0000 7f95f0ec5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.099+0000 7f95f0ec5640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.099+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d8003520 con 0x7f95ec142a20 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510729927 0 0) 0x7f95ec0a9d60 con 0x7f95ec0a96b0 2024-03-25T18:19:42.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec13ff10 con 0x7f95ec0a96b0 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600583309 0 0) 0x7f95ec0aa1f0 con 0x7f95ec0b4700 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:19:42.104 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.100+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec0a9d60 con 0x7f95ec0b4700 2024-03-25T18:19:42.105 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1511349338 0 0) 0x7f95d8003520 con 0x7f95ec142a20 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 >> v1:172.21.15.42:6790/0 conn(0x7f95ec0b4700 legacy=0x7f95ec142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 >> v1:172.21.15.42:6789/0 conn(0x7f95ec0a96b0 legacy=0x7f95ec13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 10 monclient: _finish_hunting 0 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 1 monclient: found mon.noname-c 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0b4ae0 con 0x7f95ec142a20 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 10 monclient: _finish_auth 0 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 10 monclient: _check_auth_tickets 2024-03-25T18:19:42.106 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f95dc001690 con 0x7f95ec142a20 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f0ec5640 10 monclient: dump: 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.107 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.108 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.108 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.108 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.108 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.108 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f412b640 5 monclient: authenticate success, global_id 4120 2024-03-25T18:19:42.108 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.103+0000 7f95f412b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:42.124 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.123+0000 7f31ee442040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95dc0017f0 con 0x7f95ec142a20 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f0ec5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f0ec5640 1 -- 172.21.15.120:0/1953000924 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f95dc001c00 con 0x7f95ec142a20 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f412b640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f412b640 4 set_mon_vals no callback set 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f0ec5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f0ec5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.129+0000 7f95f0ec5640 10 monclient: dump: 2024-03-25T18:19:42.132 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.130+0000 7f95f412b640 10 monclient: shutdown 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.130+0000 7f95f412b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.130+0000 7f95f412b640 1 -- 172.21.15.120:0/1953000924 >> v1:172.21.15.120:6789/0 conn(0x7f95ec142a20 legacy=0x7f95ec13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.130+0000 7f95f412b640 1 -- 172.21.15.120:0/1953000924 shutdown_connections 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.130+0000 7f95f412b640 1 -- 172.21.15.120:0/1953000924 >> 172.21.15.120:0/1953000924 conn(0x7f95ec0668e0 msgr2=0x7f95ec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.130+0000 7f95f412b640 1 -- 172.21.15.120:0/1953000924 shutdown_connections 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.130+0000 7f95f412b640 1 -- 172.21.15.120:0/1953000924 wait complete. 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 10 monclient: build_initial_monmap 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 10 monclient: monmap: 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:42.134 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 1 librados: starting msgr at 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 1 librados: starting objecter 2024-03-25T18:19:42.135 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 1 Processor -- start 2024-03-25T18:19:42.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 1 -- start start 2024-03-25T18:19:42.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 1 librados: setting wanted keys 2024-03-25T18:19:42.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 1 librados: calling monclient init 2024-03-25T18:19:42.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 10 monclient: init 2024-03-25T18:19:42.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:42.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:42.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec0a96b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:42.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.131+0000 7f95f412b640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec0b4700 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:42.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f412b640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:42.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f412b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec1db720 con 0x7f95ec142a20 2024-03-25T18:19:42.139 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f412b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec1da810 con 0x7f95ec0b4700 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f412b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec1daaf0 con 0x7f95ec0a96b0 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f412b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f412b640 10 monclient(hunting): authenticate will time out at 7300.287952s 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f1ec7640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f95ec0a96b0 0x7f95ec13b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.120:59352/0 (socket says 172.21.15.120:59352) 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95f1ec7640 1 -- 172.21.15.120:0/2393497351 learned_addr learned my addr 172.21.15.120:0/2393497351 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95e27fc640 10 client.?.objecter ms_handle_connect 0x7f95ec142a20 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.132+0000 7f95e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.133+0000 7f95e27fc640 10 client.?.objecter ms_handle_connect 0x7f95ec0a96b0 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.133+0000 7f95e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.134+0000 7f95e27fc640 10 client.?.objecter ms_handle_connect 0x7f95ec0b4700 2024-03-25T18:19:42.140 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.134+0000 7f95e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279727401 0 0) 0x7f95ec1db720 con 0x7f95ec142a20 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d0004eb0 con 0x7f95ec142a20 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215200776 0 0) 0x7f95ec1daaf0 con 0x7f95ec0a96b0 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.141 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:19:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.135+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec1db720 con 0x7f95ec0a96b0 2024-03-25T18:19:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.136+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073528084 0 0) 0x7f95ec1da810 con 0x7f95ec0b4700 2024-03-25T18:19:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.136+0000 7f95e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.136+0000 7f95e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.136+0000 7f95e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.136+0000 7f95e27fc640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:19:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.136+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec1daaf0 con 0x7f95ec0b4700 2024-03-25T18:19:42.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.140+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 722008920 0 0) 0x7f95d0004eb0 con 0x7f95ec142a20 2024-03-25T18:19:42.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.140+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95ec1da810 con 0x7f95ec142a20 2024-03-25T18:19:42.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.140+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f95e4001bd0 con 0x7f95ec142a20 2024-03-25T18:19:42.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.140+0000 7f95e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.140+0000 7f95e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:42.143 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.140+0000 7f95e27fc640 10 monclient(hunting): dump: 2024-03-25T18:19:42.143 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.144 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.140+0000 7f95e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2947697251 0 0) 0x7f95ec1da810 con 0x7f95ec142a20 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 >> v1:172.21.15.42:6790/0 conn(0x7f95ec0a96b0 legacy=0x7f95ec13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 >> v1:172.21.15.120:6789/0 conn(0x7f95ec0b4700 legacy=0x7f95ec1db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 1 monclient: found mon.a 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.146 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec134880 con 0x7f95ec142a20 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 10 monclient: _renew_subs 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec138970 con 0x7f95ec142a20 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 10 monclient: _renew_subs 2024-03-25T18:19:42.147 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec13c360 con 0x7f95ec142a20 2024-03-25T18:19:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.143+0000 7f95f412b640 1 librados: init done 2024-03-25T18:19:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.144+0000 7f95f412b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.144+0000 7f95f412b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.144+0000 7f95f412b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.148 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.144+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec0ac1c0 con 0x7f95ec142a20 2024-03-25T18:19:42.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e4003970 con 0x7f95ec142a20 2024-03-25T18:19:42.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:42.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f95e4002100 con 0x7f95ec142a20 2024-03-25T18:19:42.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:42.149 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 10 monclient: dump: 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.150 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f95e4004050 con 0x7f95ec142a20 2024-03-25T18:19:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e3fff640 4 set_mon_vals no callback set 2024-03-25T18:19:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3747586647 0 0) 0x7f95e40055c0 con 0x7f95ec142a20 2024-03-25T18:19:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 10 client.4125.objecter ms_dispatch 0x7f95ec000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:19:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:19:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:19:42.151 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.147+0000 7f95e27fc640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:19:42.169 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.168+0000 7f31ee442040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:19:42.223 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.222+0000 7f8cc6e3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:19:42.250 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.249+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f95e40059e0 con 0x7f95ec142a20 2024-03-25T18:19:42.250 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.249+0000 7f95e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.250 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.249+0000 7f95e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:42.286 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.285+0000 7f31ee442040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:19:42.297 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.296+0000 7f8cc6e3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:42.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.321+0000 7f95f412b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "76fae7c0-88cd-4858-898c-ee21e1b87701", "id": 1}] 2024-03-25T18:19:42.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.321+0000 7f95f412b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "76fae7c0-88cd-4858-898c-ee21e1b87701", "id": 1}] 2024-03-25T18:19:42.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.321+0000 7f95f412b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.321+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd new", "uuid": "76fae7c0-88cd-4858-898c-ee21e1b87701", "id": 1} v 0) v1 -- 0x7f95ec04f730 con 0x7f95ec142a20 2024-03-25T18:19:42.345 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.344+0000 7f8cc6e3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:19:42.357 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.356+0000 7f31ee442040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:42.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.360+0000 7f95e27fc640 1 -- 172.21.15.120:0/2393497351 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76fae7c0-88cd-4858-898c-ee21e1b87701", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2143980564 0 940483276) 0x7f95e4005b80 con 0x7f95ec142a20 2024-03-25T18:19:42.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.360+0000 7f95e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "76fae7c0-88cd-4858-898c-ee21e1b87701", "id": 1}] 2024-03-25T18:19:42.361 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.360+0000 7f95e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:42.362 INFO:teuthology.orchestra.run.smithi120.stdout:1 2024-03-25T18:19:42.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 10 librados: watch_flush enter 2024-03-25T18:19:42.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 10 librados: watch_flush exit 2024-03-25T18:19:42.363 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 10 monclient: shutdown 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 >> v1:172.21.15.42:6789/0 conn(0x7f95ec142a20 legacy=0x7f95ec134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 shutdown_connections 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 >> 172.21.15.120:0/2393497351 conn(0x7f95ec0668e0 msgr2=0x7f95ec0b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 shutdown_connections 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 1 -- 172.21.15.120:0/2393497351 wait complete. 2024-03-25T18:19:42.364 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.361+0000 7f95f412b640 1 librados: shutdown 2024-03-25T18:19:42.373 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new a44eba7f-b7c9-4441-8561-ef78dafca07b 2 2024-03-25T18:19:42.402 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.401+0000 7f8cc6e3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:19:42.406 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.405+0000 7f31ee442040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:19:42.451 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.451+0000 7f8cc6e3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:19:42.461 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.460+0000 7f31ee442040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:19:42.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.505+0000 7fe5701cf640 10 monclient: get_monmap_and_config 2024-03-25T18:19:42.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.505+0000 7fe5701cf640 10 monclient: build_initial_monmap 2024-03-25T18:19:42.506 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.505+0000 7fe5701cf640 10 monclient: monmap: 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:42.506299+0000 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:42.506299+0000 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.505+0000 7fe5701cf640 1 Processor -- start 2024-03-25T18:19:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 1 -- start start 2024-03-25T18:19:42.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient: init 2024-03-25T18:19:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe5680b4700 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe568142a20 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:42.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe5680a96b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680a9d60 con 0x7fe5680b4700 2024-03-25T18:19:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56813ff10 con 0x7fe568142a20 2024-03-25T18:19:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680aa1f0 con 0x7fe5680a96b0 2024-03-25T18:19:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.506+0000 7fe5701cf640 10 monclient(hunting): authenticate will time out at 7300.662598s 2024-03-25T18:19:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.507+0000 7fe56d76a640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe568142a20 0x7fe56813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47402/0 (socket says 172.21.15.120:47402) 2024-03-25T18:19:42.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.507+0000 7fe56d76a640 1 -- 172.21.15.120:0/1930542701 learned_addr learned my addr 172.21.15.120:0/1930542701 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.509+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887442083 0 0) 0x7fe56813ff10 con 0x7fe568142a20 2024-03-25T18:19:42.511 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.509+0000 7fe56cf69640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.509+0000 7fe56cf69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.509+0000 7fe56cf69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.509+0000 7fe56cf69640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.509+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe554003520 con 0x7fe568142a20 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414043413 0 0) 0x7fe5680aa1f0 con 0x7fe5680a96b0 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe56813ff10 con 0x7fe5680a96b0 2024-03-25T18:19:42.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328998047 0 0) 0x7fe5680a9d60 con 0x7fe5680b4700 2024-03-25T18:19:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:19:42.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.510+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5680aa1f0 con 0x7fe5680b4700 2024-03-25T18:19:42.513 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.510+0000 7f31ee442040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:19:42.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956579279 0 0) 0x7fe554003520 con 0x7fe568142a20 2024-03-25T18:19:42.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 >> v1:172.21.15.42:6790/0 conn(0x7fe5680a96b0 legacy=0x7fe56813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 >> v1:172.21.15.42:6789/0 conn(0x7fe5680b4700 legacy=0x7fe568142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 10 monclient: _finish_hunting 0 2024-03-25T18:19:42.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 1 monclient: found mon.noname-c 2024-03-25T18:19:42.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:42.514 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680b4ae0 con 0x7fe568142a20 2024-03-25T18:19:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 10 monclient: _finish_auth 0 2024-03-25T18:19:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 10 monclient: _check_auth_tickets 2024-03-25T18:19:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe56cf69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe5701cf640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:19:42.515 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.512+0000 7fe5701cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:42.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe558001650 con 0x7fe568142a20 2024-03-25T18:19:42.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:42.517 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 10 monclient: dump: 2024-03-25T18:19:42.517 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.517 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.518 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5580018a0 con 0x7fe568142a20 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 1 -- 172.21.15.120:0/1930542701 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe558001c30 con 0x7fe568142a20 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe5701cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe5701cf640 4 set_mon_vals no callback set 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.514+0000 7fe56cf69640 10 monclient: dump: 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.519 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.515+0000 7fe5701cf640 10 monclient: shutdown 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.515+0000 7fe5701cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.515+0000 7fe5701cf640 1 -- 172.21.15.120:0/1930542701 >> v1:172.21.15.120:6789/0 conn(0x7fe568142a20 legacy=0x7fe56813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.515+0000 7fe5701cf640 1 -- 172.21.15.120:0/1930542701 shutdown_connections 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.515+0000 7fe5701cf640 1 -- 172.21.15.120:0/1930542701 >> 172.21.15.120:0/1930542701 conn(0x7fe5680668e0 msgr2=0x7fe568066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:42.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.515+0000 7fe5701cf640 1 -- 172.21.15.120:0/1930542701 shutdown_connections 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.515+0000 7fe5701cf640 1 -- 172.21.15.120:0/1930542701 wait complete. 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 10 monclient: build_initial_monmap 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 10 monclient: monmap: 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:42.521 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 1 librados: starting msgr at 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 1 librados: starting objecter 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 1 Processor -- start 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 1 -- start start 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 1 librados: setting wanted keys 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 1 librados: calling monclient init 2024-03-25T18:19:42.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.516+0000 7fe5701cf640 10 monclient: init 2024-03-25T18:19:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe5680a96b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:42.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe5680b4700 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe568142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5681d7460 con 0x7fe568142a20 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680aded0 con 0x7fe5680b4700 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680acb40 con 0x7fe5680a96b0 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe5701cf640 10 monclient(hunting): authenticate will time out at 7300.673292s 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe56d76a640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe5680b4700 0x7fe5681d7bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47410/0 (socket says 172.21.15.120:47410) 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.517+0000 7fe56d76a640 1 -- 172.21.15.120:0/2338961991 learned_addr learned my addr 172.21.15.120:0/2338961991 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.518+0000 7fe55e7fc640 10 client.?.objecter ms_handle_connect 0x7fe5680b4700 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.518+0000 7fe55e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.518+0000 7fe55e7fc640 10 client.?.objecter ms_handle_connect 0x7fe568142a20 2024-03-25T18:19:42.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.518+0000 7fe55e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.518+0000 7fe55e7fc640 10 client.?.objecter ms_handle_connect 0x7fe5680a96b0 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.518+0000 7fe55e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.519+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228967687 0 0) 0x7fe5680aded0 con 0x7fe5680b4700 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.519+0000 7fe55e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.519+0000 7fe55e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.519+0000 7fe55e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.519+0000 7fe55e7fc640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.519+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54c004eb0 con 0x7fe5680b4700 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014743014 0 0) 0x7fe5681d7460 con 0x7fe568142a20 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5680aded0 con 0x7fe568142a20 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241646 0 0) 0x7fe5680acb40 con 0x7fe5680a96b0 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.520+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5681d7460 con 0x7fe5680a96b0 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.524+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4196155337 0 0) 0x7fe54c004eb0 con 0x7fe5680b4700 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.524+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5680acb40 con 0x7fe5680b4700 2024-03-25T18:19:42.529 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.524+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe558001c90 con 0x7fe5680b4700 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.524+0000 7fe55e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.524+0000 7fe55e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.524+0000 7fe55e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.530 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.524+0000 7fe55e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1199179213 0 0) 0x7fe5680aded0 con 0x7fe568142a20 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe54c004eb0 con 0x7fe568142a20 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe560001b50 con 0x7fe568142a20 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.531 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3863494942 0 0) 0x7fe5681d7460 con 0x7fe5680a96b0 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5680aded0 con 0x7fe5680a96b0 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe564001870 con 0x7fe5680a96b0 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:42.532 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.525+0000 7fe55e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.533 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3165946445 0 0) 0x7fe54c004eb0 con 0x7fe568142a20 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 >> v1:172.21.15.42:6790/0 conn(0x7fe5680a96b0 legacy=0x7fe5680ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 >> v1:172.21.15.120:6789/0 conn(0x7fe5680b4700 legacy=0x7fe5681d7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 1 monclient: found mon.a 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5681d82e0 con 0x7fe568142a20 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:42.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe55e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 10 monclient: _renew_subs 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5681d76b0 con 0x7fe568142a20 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 10 monclient: _renew_subs 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5680acf90 con 0x7fe568142a20 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.529+0000 7fe5701cf640 1 librados: init done 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.530+0000 7fe5701cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.530+0000 7fe5701cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.530+0000 7fe5701cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.530+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5681d78f0 con 0x7fe568142a20 2024-03-25T18:19:42.639 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe560002060 con 0x7fe568142a20 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe560003970 con 0x7fe568142a20 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55e7fc640 10 monclient: dump: 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.640 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe560003e60 con 0x7fe568142a20 2024-03-25T18:19:42.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.638+0000 7fe55ffff640 4 set_mon_vals no callback set 2024-03-25T18:19:42.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.639+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 4212395343 0 0) 0x7fe560005440 con 0x7fe568142a20 2024-03-25T18:19:42.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.639+0000 7fe55e7fc640 10 client.4131.objecter ms_dispatch 0x7fe568000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:19:42.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.639+0000 7fe55e7fc640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:19:42.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.639+0000 7fe55e7fc640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:19:42.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.640+0000 7fe55e7fc640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:19:42.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.640+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe560005020 con 0x7fe568142a20 2024-03-25T18:19:42.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.640+0000 7fe55e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.640+0000 7fe55e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:42.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.735+0000 7fe5701cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a44eba7f-b7c9-4441-8561-ef78dafca07b", "id": 2}] 2024-03-25T18:19:42.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.735+0000 7fe5701cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a44eba7f-b7c9-4441-8561-ef78dafca07b", "id": 2}] 2024-03-25T18:19:42.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.735+0000 7fe5701cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:42.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.735+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a44eba7f-b7c9-4441-8561-ef78dafca07b", "id": 2} v 0) v1 -- 0x7fe56804f730 con 0x7fe568142a20 2024-03-25T18:19:42.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.766+0000 7fe55e7fc640 1 -- 172.21.15.120:0/2338961991 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a44eba7f-b7c9-4441-8561-ef78dafca07b", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1566051982 0 216599125) 0x7fe5600052a0 con 0x7fe568142a20 2024-03-25T18:19:42.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.766+0000 7fe55e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a44eba7f-b7c9-4441-8561-ef78dafca07b", "id": 2}] 2024-03-25T18:19:42.767 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.766+0000 7fe55e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:42.768 INFO:teuthology.orchestra.run.smithi120.stdout:2 2024-03-25T18:19:42.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 10 librados: watch_flush enter 2024-03-25T18:19:42.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 10 librados: watch_flush exit 2024-03-25T18:19:42.769 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 10 monclient: shutdown 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 >> v1:172.21.15.42:6789/0 conn(0x7fe568142a20 legacy=0x7fe5681e1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 shutdown_connections 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 >> 172.21.15.120:0/2338961991 conn(0x7fe5680668e0 msgr2=0x7fe5680b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 shutdown_connections 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 1 -- 172.21.15.120:0/2338961991 wait complete. 2024-03-25T18:19:42.770 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.767+0000 7fe5701cf640 1 librados: shutdown 2024-03-25T18:19:42.780 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 408e355a-5bb9-4fde-8fb3-c5256132c2ee 3 2024-03-25T18:19:42.854 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.854+0000 7f8cc6e3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:19:42.879 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.878+0000 7f9fe2e31640 10 monclient: get_monmap_and_config 2024-03-25T18:19:42.879 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.878+0000 7f9fe2e31640 10 monclient: build_initial_monmap 2024-03-25T18:19:42.879 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.878+0000 7f9fe2e31640 10 monclient: monmap: 2024-03-25T18:19:42.879 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:42.879725+0000 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:42.879725+0000 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:42.880 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.879+0000 7f9fe2e31640 1 Processor -- start 2024-03-25T18:19:42.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.879+0000 7f9fe2e31640 1 -- start start 2024-03-25T18:19:42.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.879+0000 7f9fe2e31640 10 monclient: init 2024-03-25T18:19:42.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:42.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:42.882 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 10 monclient(hunting): picked mon.noname-c con 0x7f9fdc0b4700 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 10 monclient(hunting): picked mon.noname-a con 0x7f9fdc142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 10 monclient(hunting): picked mon.noname-b con 0x7f9fdc0a96b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc0a9d60 con 0x7f9fdc142a20 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc13ff10 con 0x7f9fdc0b4700 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc0aa1f0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe2e31640 10 monclient(hunting): authenticate will time out at 7301.036021s 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe0bcd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9fdc0b4700 0x7f9fdc142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47426/0 (socket says 172.21.15.120:47426) 2024-03-25T18:19:42.883 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.880+0000 7f9fe0bcd640 1 -- 172.21.15.120:0/4183380307 learned_addr learned my addr 172.21.15.120:0/4183380307 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:42.884 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.882+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598516335 0 0) 0x7f9fdc13ff10 con 0x7f9fdc0b4700 2024-03-25T18:19:42.884 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.882+0000 7f9fdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.882+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.882+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.882+0000 7f9fdb7fe640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.882+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fcc003520 con 0x7f9fdc0b4700 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238415020 0 0) 0x7f9fdc0aa1f0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:19:42.885 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fdc13ff10 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.886 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68809670 0 0) 0x7f9fdc0a9d60 con 0x7f9fdc142a20 2024-03-25T18:19:42.886 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.886 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.886 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.886 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:19:42.886 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.883+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fdc0aa1f0 con 0x7f9fdc142a20 2024-03-25T18:19:42.887 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1633193896 0 0) 0x7f9fcc003520 con 0x7f9fdc0b4700 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 >> v1:172.21.15.42:6790/0 conn(0x7f9fdc0a96b0 legacy=0x7f9fdc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 >> v1:172.21.15.42:6789/0 conn(0x7f9fdc142a20 legacy=0x7f9fdc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc0b4ae0 con 0x7f9fdc0b4700 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:42.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fe2e31640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.885+0000 7f9fe2e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.886+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9fd0001650 con 0x7f9fdc0b4700 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.886+0000 7f9fdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.886+0000 7f9fdb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.886+0000 7f9fdb7fe640 10 monclient: dump: 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.889 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.886+0000 7f9fe2e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:42.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fd00018a0 con 0x7f9fdc0b4700 2024-03-25T18:19:42.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:42.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fdb7fe640 1 -- 172.21.15.120:0/4183380307 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9fd0001c30 con 0x7f9fdc0b4700 2024-03-25T18:19:42.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fdb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:42.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fdb7fe640 10 monclient: dump: 2024-03-25T18:19:42.894 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fe2e31640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:42.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.887+0000 7f9fe2e31640 4 set_mon_vals no callback set 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.888+0000 7f9fe2e31640 10 monclient: shutdown 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.888+0000 7f9fe2e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.888+0000 7f9fe2e31640 1 -- 172.21.15.120:0/4183380307 >> v1:172.21.15.120:6789/0 conn(0x7f9fdc0b4700 legacy=0x7f9fdc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.888+0000 7f9fe2e31640 1 -- 172.21.15.120:0/4183380307 shutdown_connections 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.888+0000 7f9fe2e31640 1 -- 172.21.15.120:0/4183380307 >> 172.21.15.120:0/4183380307 conn(0x7f9fdc0668e0 msgr2=0x7f9fdc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.888+0000 7f9fe2e31640 1 -- 172.21.15.120:0/4183380307 shutdown_connections 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.888+0000 7f9fe2e31640 1 -- 172.21.15.120:0/4183380307 wait complete. 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 10 monclient: build_initial_monmap 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 10 monclient: monmap: 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:42.896 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 1 librados: starting msgr at 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 1 librados: starting objecter 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 1 Processor -- start 2024-03-25T18:19:42.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 1 -- start start 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 1 librados: setting wanted keys 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 1 librados: calling monclient init 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.889+0000 7f9fe2e31640 10 monclient: init 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 10 monclient(hunting): picked mon.noname-b con 0x7f9fdc0a96b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 10 monclient(hunting): picked mon.noname-c con 0x7f9fdc0b4700 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 10 monclient(hunting): picked mon.noname-a con 0x7f9fdc142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc0ac0e0 con 0x7f9fdc142a20 2024-03-25T18:19:42.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc0b1830 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc0acb90 con 0x7f9fdc0b4700 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe2e31640 10 monclient(hunting): authenticate will time out at 7301.046452s 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe0bcd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9fdc0a96b0 0x7f9fdc0ad850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47430/0 (socket says 172.21.15.120:47430) 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.890+0000 7f9fe0bcd640 1 -- 172.21.15.120:0/1743016938 learned_addr learned my addr 172.21.15.120:0/1743016938 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.891+0000 7f9fd97fa640 10 client.?.objecter ms_handle_connect 0x7f9fdc0a96b0 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.891+0000 7f9fd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.891+0000 7f9fd97fa640 10 client.?.objecter ms_handle_connect 0x7f9fdc0b4700 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.891+0000 7f9fd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.899 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.891+0000 7f9fd97fa640 10 client.?.objecter ms_handle_connect 0x7f9fdc142a20 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.891+0000 7f9fd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501984522 0 0) 0x7f9fdc0b1830 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc0004eb0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489331254 0 0) 0x7f9fdc0acb90 con 0x7f9fdc0b4700 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fdc0b1830 con 0x7f9fdc0b4700 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344802962 0 0) 0x7f9fdc0ac0e0 con 0x7f9fdc142a20 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:19:42.901 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.893+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fdc0acb90 con 0x7f9fdc142a20 2024-03-25T18:19:42.903 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.897+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1591056906 0 0) 0x7f9fc0004eb0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.897+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fdc0ac0e0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.897+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9fd0001bb0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.897+0000 7f9fd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.897+0000 7f9fd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.897+0000 7f9fd97fa640 10 monclient(hunting): dump: 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.904 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.897+0000 7f9fd97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3634903377 0 0) 0x7f9fdc0acb90 con 0x7f9fdc142a20 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fc0004eb0 con 0x7f9fdc142a20 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9fd4001a70 con 0x7f9fdc142a20 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:42.905 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 10 monclient(hunting): dump: 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 127661318 0 0) 0x7f9fdc0b1830 con 0x7f9fdc0b4700 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fdc0acb90 con 0x7f9fdc0b4700 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9fc8001870 con 0x7f9fdc0b4700 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.898+0000 7f9fd97fa640 10 monclient(hunting): dump: 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.907 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1918468661 0 0) 0x7f9fdc0ac0e0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 >> v1:172.21.15.42:6790/0 conn(0x7f9fdc0b4700 legacy=0x7f9fdc139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 >> v1:172.21.15.42:6789/0 conn(0x7f9fdc142a20 legacy=0x7f9fdc1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 1 monclient: found mon.b 2024-03-25T18:19:42.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc0ace70 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 10 monclient: _renew_subs 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fdc0ad150 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 10 monclient: _renew_subs 2024-03-25T18:19:42.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc13bce0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.901+0000 7f9fe2e31640 1 librados: init done 2024-03-25T18:19:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.902+0000 7f9fe2e31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.902+0000 7f9fe2e31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.902+0000 7f9fe2e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:42.910 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.902+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc04f730 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.923 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:42.923+0000 7f31ee442040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fd0004bf0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9fd0004f00 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 10 monclient: dump: 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:42.927 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9fd00053f0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 74140097 0 0) 0x7f9fd00069b0 con 0x7f9fdc0a96b0 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fdaffd640 4 set_mon_vals no callback set 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 10 client.4135.objecter ms_dispatch 0x7f9fdc000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:19:42.928 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:42.925+0000 7f9fd97fa640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-25T18:19:42.999 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:42.998+0000 7f8cc6e3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:19:43.061 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.060+0000 7f31ee442040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:19:43.095 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.095+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9fd00065d0 con 0x7f9fdc0a96b0 2024-03-25T18:19:43.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.095+0000 7f9fd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.096 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.095+0000 7f9fd97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:43.130 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:43.130+0000 7f8cc6e3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:19:43.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.168+0000 7f9fe2e31640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "408e355a-5bb9-4fde-8fb3-c5256132c2ee", "id": 3}] 2024-03-25T18:19:43.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.168+0000 7f9fe2e31640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "408e355a-5bb9-4fde-8fb3-c5256132c2ee", "id": 3}] 2024-03-25T18:19:43.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.168+0000 7f9fe2e31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.169 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.168+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd new", "uuid": "408e355a-5bb9-4fde-8fb3-c5256132c2ee", "id": 3} v 0) v1 -- 0x7f9fdc0b7190 con 0x7f9fdc0a96b0 2024-03-25T18:19:43.178 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:43.177+0000 7f8cc6e3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:19:43.192 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.191+0000 7f31ee442040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:19:43.252 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.251+0000 7f31ee442040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:19:43.298 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:43.297+0000 7f8cc6e3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:43.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.303+0000 7f9fd97fa640 1 -- 172.21.15.120:0/1743016938 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "408e355a-5bb9-4fde-8fb3-c5256132c2ee", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3755695037 0 525046306) 0x7f9fd0006bc0 con 0x7f9fdc0a96b0 2024-03-25T18:19:43.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.303+0000 7f9fd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "408e355a-5bb9-4fde-8fb3-c5256132c2ee", "id": 3}] 2024-03-25T18:19:43.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.303+0000 7f9fd97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:43.304 INFO:teuthology.orchestra.run.smithi120.stdout:3 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 10 librados: watch_flush enter 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 10 librados: watch_flush exit 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 10 client.4135.objecter successfully canceled tick 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 10 monclient: shutdown 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 >> v1:172.21.15.120:6789/0 conn(0x7f9fdc0a96b0 legacy=0x7f9fdc0ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 shutdown_connections 2024-03-25T18:19:43.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 >> 172.21.15.120:0/1743016938 conn(0x7f9fdc0668e0 msgr2=0x7f9fdc1364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:43.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 shutdown_connections 2024-03-25T18:19:43.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 1 -- 172.21.15.120:0/1743016938 wait complete. 2024-03-25T18:19:43.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.304+0000 7f9fe2e31640 1 librados: shutdown 2024-03-25T18:19:43.316 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 8e53f94b-2a33-459d-b200-fd4fe94ea86c 4 2024-03-25T18:19:43.386 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.385+0000 7f31ee442040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:43.397 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:43.396+0000 7f8cc6e3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:19:43.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.414+0000 7f7b9813b640 10 monclient: get_monmap_and_config 2024-03-25T18:19:43.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.414+0000 7f7b9813b640 10 monclient: build_initial_monmap 2024-03-25T18:19:43.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.414+0000 7f7b9813b640 10 monclient: monmap: 2024-03-25T18:19:43.415 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:43.415294+0000 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:43.415294+0000 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.414+0000 7f7b9813b640 1 Processor -- start 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 1 -- start start 2024-03-25T18:19:43.416 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient: init 2024-03-25T18:19:43.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:43.417 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b900b4700 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b90142a20 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b900a96b0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900a9d60 con 0x7f7b900a96b0 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9013ff10 con 0x7f7b90142a20 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900aa1f0 con 0x7f7b900b4700 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.415+0000 7f7b9813b640 10 monclient(hunting): authenticate will time out at 7301.571582s 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.416+0000 7f7b956d6640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f7b90142a20 0x7f7b9013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47438/0 (socket says 172.21.15.120:47438) 2024-03-25T18:19:43.418 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.416+0000 7f7b956d6640 1 -- 172.21.15.120:0/98713997 learned_addr learned my addr 172.21.15.120:0/98713997 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:43.419 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996520396 0 0) 0x7f7b9013ff10 con 0x7f7b90142a20 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c003520 con 0x7f7b90142a20 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356096367 0 0) 0x7f7b900a9d60 con 0x7f7b900a96b0 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9013ff10 con 0x7f7b900a96b0 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996587977 0 0) 0x7f7b900aa1f0 con 0x7f7b900b4700 2024-03-25T18:19:43.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:19:43.421 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.418+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b900a9d60 con 0x7f7b900b4700 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.421+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389475820 0 0) 0x7f7b7c003520 con 0x7f7b90142a20 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 >> v1:172.21.15.42:6790/0 conn(0x7f7b900b4700 legacy=0x7f7b90142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 >> v1:172.21.15.42:6789/0 conn(0x7f7b900a96b0 legacy=0x7f7b9013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 10 monclient: _finish_hunting 0 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 1 monclient: found mon.noname-c 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.423 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b900b4ae0 con 0x7f7b90142a20 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 10 monclient: _finish_auth 0 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 10 monclient: _check_auth_tickets 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7b80001690 con 0x7f7b90142a20 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b94ed5640 10 monclient: dump: 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.424 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.425 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.425 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.425 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b9813b640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:19:43.425 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.422+0000 7f7b9813b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b800017f0 con 0x7f7b90142a20 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b94ed5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b94ed5640 1 -- 172.21.15.120:0/98713997 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7b80001c00 con 0x7f7b90142a20 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b94ed5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b94ed5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b94ed5640 10 monclient: dump: 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.428 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b9813b640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.423+0000 7f7b9813b640 4 set_mon_vals no callback set 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.424+0000 7f7b9813b640 10 monclient: shutdown 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.424+0000 7f7b9813b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.424+0000 7f7b9813b640 1 -- 172.21.15.120:0/98713997 >> v1:172.21.15.120:6789/0 conn(0x7f7b90142a20 legacy=0x7f7b9013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.424+0000 7f7b9813b640 1 -- 172.21.15.120:0/98713997 shutdown_connections 2024-03-25T18:19:43.429 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.424+0000 7f7b9813b640 1 -- 172.21.15.120:0/98713997 >> 172.21.15.120:0/98713997 conn(0x7f7b900668e0 msgr2=0x7f7b90066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.424+0000 7f7b9813b640 1 -- 172.21.15.120:0/98713997 shutdown_connections 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.424+0000 7f7b9813b640 1 -- 172.21.15.120:0/98713997 wait complete. 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 10 monclient: build_initial_monmap 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 10 monclient: monmap: 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:43.430 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 1 librados: starting msgr at 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 1 librados: starting objecter 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 1 Processor -- start 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 1 -- start start 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 1 librados: setting wanted keys 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 1 librados: calling monclient init 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.425+0000 7f7b9813b640 10 monclient: init 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b900a96b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b900b4700 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b90142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b901db720 con 0x7f7b90142a20 2024-03-25T18:19:43.431 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b901da810 con 0x7f7b900a96b0 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b901daaf0 con 0x7f7b900b4700 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b9813b640 10 monclient(hunting): authenticate will time out at 7301.582236s 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b95ed7640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f7b900a96b0 0x7f7b9013b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47446/0 (socket says 172.21.15.120:47446) 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b95ed7640 1 -- 172.21.15.120:0/1996880695 learned_addr learned my addr 172.21.15.120:0/1996880695 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b867fc640 10 client.?.objecter ms_handle_connect 0x7f7b900a96b0 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.426+0000 7f7b867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.427+0000 7f7b867fc640 10 client.?.objecter ms_handle_connect 0x7f7b90142a20 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.427+0000 7f7b867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.427+0000 7f7b867fc640 10 client.?.objecter ms_handle_connect 0x7f7b900b4700 2024-03-25T18:19:43.432 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.427+0000 7f7b867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.428+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636173351 0 0) 0x7f7b901da810 con 0x7f7b900a96b0 2024-03-25T18:19:43.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.428+0000 7f7b867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.433 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.428+0000 7f7b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.428+0000 7f7b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.428+0000 7f7b867fc640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.428+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b74004eb0 con 0x7f7b900a96b0 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747521916 0 0) 0x7f7b901db720 con 0x7f7b90142a20 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b901da810 con 0x7f7b90142a20 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011573184 0 0) 0x7f7b901daaf0 con 0x7f7b900b4700 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.434 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:19:43.435 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.429+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b901db720 con 0x7f7b900b4700 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.433+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686532238 0 0) 0x7f7b74004eb0 con 0x7f7b900a96b0 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.433+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b901daaf0 con 0x7f7b900a96b0 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.433+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7b8c001a10 con 0x7f7b900a96b0 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.433+0000 7f7b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.433+0000 7f7b867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.433+0000 7f7b867fc640 10 monclient(hunting): dump: 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.436 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.433+0000 7f7b867fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441080602 0 0) 0x7f7b901db720 con 0x7f7b900b4700 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b74004eb0 con 0x7f7b900b4700 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7b80001b00 con 0x7f7b900b4700 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 10 monclient(hunting): dump: 2024-03-25T18:19:43.437 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3156725268 0 0) 0x7f7b901da810 con 0x7f7b90142a20 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b901db720 con 0x7f7b90142a20 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7b88001b50 con 0x7f7b90142a20 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.434+0000 7f7b867fc640 10 monclient(hunting): dump: 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.439 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 490370890 0 0) 0x7f7b901daaf0 con 0x7f7b900a96b0 2024-03-25T18:19:43.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 >> v1:172.21.15.42:6790/0 conn(0x7f7b900b4700 legacy=0x7f7b901db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 >> v1:172.21.15.42:6789/0 conn(0x7f7b90142a20 legacy=0x7f7b90134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 1 monclient: found mon.b 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b90134880 con 0x7f7b900a96b0 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 10 monclient: _renew_subs 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b90138970 con 0x7f7b900a96b0 2024-03-25T18:19:43.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 10 monclient: _renew_subs 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9013c360 con 0x7f7b900a96b0 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 1 librados: init done 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.437+0000 7f7b9813b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.438+0000 7f7b9813b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.438+0000 7f7b9813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.438+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b900ac1c0 con 0x7f7b900a96b0 2024-03-25T18:19:43.486 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.485+0000 7f31ee442040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:19:43.535 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c001c70 con 0x7f7b900a96b0 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7b8c003a60 con 0x7f7b900a96b0 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 10 monclient: dump: 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.536 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7b8c004010 con 0x7f7b900a96b0 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b87fff640 4 set_mon_vals no callback set 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 562997817 0 0) 0x7f7b8c0067d0 con 0x7f7b900a96b0 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 10 client.4141.objecter ms_dispatch 0x7f7b90000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7b8c006b30 con 0x7f7b900a96b0 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.537 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.534+0000 7f7b867fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:43.560 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:43.559+0000 7f8cc6e3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:43.603 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:43.603+0000 7f8cc6e3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:43.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.607+0000 7f7b9813b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8e53f94b-2a33-459d-b200-fd4fe94ea86c", "id": 4}] 2024-03-25T18:19:43.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.607+0000 7f7b9813b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8e53f94b-2a33-459d-b200-fd4fe94ea86c", "id": 4}] 2024-03-25T18:19:43.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.607+0000 7f7b9813b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.608 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.607+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8e53f94b-2a33-459d-b200-fd4fe94ea86c", "id": 4} v 0) v1 -- 0x7f7b9004f730 con 0x7f7b900a96b0 2024-03-25T18:19:43.645 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.644+0000 7f31ee442040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:43.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.668+0000 7f7b867fc640 1 -- 172.21.15.120:0/1996880695 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8e53f94b-2a33-459d-b200-fd4fe94ea86c", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3961964123 0 1697012583) 0x7f7b8c0062c0 con 0x7f7b900a96b0 2024-03-25T18:19:43.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.668+0000 7f7b867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8e53f94b-2a33-459d-b200-fd4fe94ea86c", "id": 4}] 2024-03-25T18:19:43.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.668+0000 7f7b867fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:43.669 INFO:teuthology.orchestra.run.smithi120.stdout:4 2024-03-25T18:19:43.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.668+0000 7f7b9813b640 10 librados: watch_flush enter 2024-03-25T18:19:43.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 10 librados: watch_flush exit 2024-03-25T18:19:43.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T18:19:43.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 10 client.4141.objecter successfully canceled tick 2024-03-25T18:19:43.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 10 monclient: shutdown 2024-03-25T18:19:43.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:43.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 >> v1:172.21.15.120:6789/0 conn(0x7f7b900a96b0 legacy=0x7f7b9013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 shutdown_connections 2024-03-25T18:19:43.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 >> 172.21.15.120:0/1996880695 conn(0x7f7b900668e0 msgr2=0x7f7b900b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:43.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 shutdown_connections 2024-03-25T18:19:43.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 1 -- 172.21.15.120:0/1996880695 wait complete. 2024-03-25T18:19:43.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.669+0000 7f7b9813b640 1 librados: shutdown 2024-03-25T18:19:43.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new d98d0642-875c-461c-80aa-aa17bdd13517 5 2024-03-25T18:19:43.690 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.689+0000 7f31ee442040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:43.769 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:43.768+0000 7f8cc6e3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.783+0000 7fe36b214640 10 monclient: get_monmap_and_config 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.783+0000 7fe36b214640 10 monclient: build_initial_monmap 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.783+0000 7fe36b214640 10 monclient: monmap: 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:43.784401+0000 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:43.784401+0000 2024-03-25T18:19:43.784 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:43.785 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.785 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:43.785 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:43.785 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:43.785 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 1 Processor -- start 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 1 -- start start 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient: init 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient(hunting): picked mon.noname-c con 0x7fe3640a9920 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient(hunting): picked mon.noname-b con 0x7fe3640b45a0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:43.786 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient(hunting): picked mon.noname-a con 0x7fe364146450 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36406b100 con 0x7fe364146450 2024-03-25T18:19:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3640b2940 con 0x7fe3640a9920 2024-03-25T18:19:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36413fec0 con 0x7fe3640b45a0 2024-03-25T18:19:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.784+0000 7fe36b214640 10 monclient(hunting): authenticate will time out at 7301.940722s 2024-03-25T18:19:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.785+0000 7fe368fb0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe3640a9920 0x7fe36413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47448/0 (socket says 172.21.15.120:47448) 2024-03-25T18:19:43.787 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.785+0000 7fe368fb0640 1 -- 172.21.15.120:0/4253310811 learned_addr learned my addr 172.21.15.120:0/4253310811 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:43.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789337595 0 0) 0x7fe3640b2940 con 0x7fe3640a9920 2024-03-25T18:19:43.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.788 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe354003520 con 0x7fe3640a9920 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174845288 0 0) 0x7fe36413fec0 con 0x7fe3640b45a0 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3640b2940 con 0x7fe3640b45a0 2024-03-25T18:19:43.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893562941 0 0) 0x7fe36406b100 con 0x7fe364146450 2024-03-25T18:19:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:19:43.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.787+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36413fec0 con 0x7fe364146450 2024-03-25T18:19:43.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 567784909 0 0) 0x7fe354003520 con 0x7fe3640a9920 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 >> v1:172.21.15.42:6790/0 conn(0x7fe3640b45a0 legacy=0x7fe364145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 >> v1:172.21.15.42:6789/0 conn(0x7fe364146450 legacy=0x7fe36414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 1 monclient: found mon.noname-c 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364140050 con 0x7fe3640a9920 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:43.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe358001690 con 0x7fe3640a9920 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe3637fe640 10 monclient: dump: 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.793 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.794 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe36b214640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:19:43.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.790+0000 7fe36b214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:43.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3580017f0 con 0x7fe3640a9920 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe3637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 4 set_mon_vals no callback set 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe3637fe640 1 -- 172.21.15.120:0/4253310811 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe358001a60 con 0x7fe3640a9920 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe3637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe3637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe3637fe640 10 monclient: dump: 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.798 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 10 monclient: shutdown 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 1 -- 172.21.15.120:0/4253310811 >> v1:172.21.15.120:6789/0 conn(0x7fe3640a9920 legacy=0x7fe36413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 1 -- 172.21.15.120:0/4253310811 shutdown_connections 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 1 -- 172.21.15.120:0/4253310811 >> 172.21.15.120:0/4253310811 conn(0x7fe3640b7f40 msgr2=0x7fe3640b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 1 -- 172.21.15.120:0/4253310811 shutdown_connections 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.792+0000 7fe36b214640 1 -- 172.21.15.120:0/4253310811 wait complete. 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 10 monclient: build_initial_monmap 2024-03-25T18:19:43.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 10 monclient: monmap: 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 1 librados: starting msgr at 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 1 librados: starting objecter 2024-03-25T18:19:43.800 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 1 Processor -- start 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 1 -- start start 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 1 librados: setting wanted keys 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 1 librados: calling monclient init 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.793+0000 7fe36b214640 10 monclient: init 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 10 monclient(hunting): picked mon.noname-c con 0x7fe3640a9920 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 10 monclient(hunting): picked mon.noname-a con 0x7fe3640b45a0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 10 monclient(hunting): picked mon.noname-b con 0x7fe364146450 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe364141c10 con 0x7fe3640b45a0 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3641425f0 con 0x7fe364146450 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3641d50b0 con 0x7fe3640a9920 2024-03-25T18:19:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.794+0000 7fe36b214640 10 monclient(hunting): authenticate will time out at 7301.950582s 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3697b1640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe364146450 0x7fe3641d49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47460/0 (socket says 172.21.15.120:47460) 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3697b1640 1 -- 172.21.15.120:0/3166372532 learned_addr learned my addr 172.21.15.120:0/3166372532 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3617fa640 10 client.?.objecter ms_handle_connect 0x7fe3640b45a0 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3617fa640 10 client.?.objecter ms_handle_connect 0x7fe364146450 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3617fa640 10 client.?.objecter ms_handle_connect 0x7fe3640a9920 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.795+0000 7fe3617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691181424 0 0) 0x7fe3641425f0 con 0x7fe364146450 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.802 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.803 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:19:43.803 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe348004eb0 con 0x7fe364146450 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369639186 0 0) 0x7fe364141c10 con 0x7fe3640b45a0 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3641425f0 con 0x7fe3640b45a0 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429118967 0 0) 0x7fe3641d50b0 con 0x7fe3640a9920 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:43.804 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.797+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe364141c10 con 0x7fe3640a9920 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036200970 0 0) 0x7fe348004eb0 con 0x7fe364146450 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3641d50b0 con 0x7fe364146450 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe35c001c70 con 0x7fe364146450 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 10 monclient(hunting): dump: 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.805 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030763912 0 0) 0x7fe364141c10 con 0x7fe3640a9920 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe348004eb0 con 0x7fe3640a9920 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe358001430 con 0x7fe3640a9920 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.802+0000 7fe3617fa640 10 monclient(hunting): dump: 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.806 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.807 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.807 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.807 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.807 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.807 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.807 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.807 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.804+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589300045 0 0) 0x7fe3641425f0 con 0x7fe3640b45a0 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.804+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe364141c10 con 0x7fe3640b45a0 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.804+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe350001970 con 0x7fe3640b45a0 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.804+0000 7fe3617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.804+0000 7fe3617fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.804+0000 7fe3617fa640 10 monclient(hunting): dump: 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.809 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3121990914 0 0) 0x7fe3641d50b0 con 0x7fe364146450 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 >> v1:172.21.15.42:6790/0 conn(0x7fe3640a9920 legacy=0x7fe3641daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 >> v1:172.21.15.42:6789/0 conn(0x7fe3640b45a0 legacy=0x7fe364142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 1 monclient: found mon.b 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:43.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364141e60 con 0x7fe364146450 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe3617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 10 monclient: _renew_subs 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe364142140 con 0x7fe364146450 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 10 monclient: _renew_subs 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36413c6b0 con 0x7fe364146450 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.805+0000 7fe36b214640 1 librados: init done 2024-03-25T18:19:43.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.806+0000 7fe36b214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.812 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.806+0000 7fe36b214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.812 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.806+0000 7fe36b214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.812 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.806+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36404f730 con 0x7fe364146450 2024-03-25T18:19:43.853 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:43.852+0000 7f31ee442040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:19:43.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe35c003990 con 0x7fe364146450 2024-03-25T18:19:43.906 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe35c003da0 con 0x7fe364146450 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 10 monclient: dump: 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:43.907 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe35c004350 con 0x7fe364146450 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe362ffd640 4 set_mon_vals no callback set 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3738039741 0 0) 0x7fe35c006ad0 con 0x7fe364146450 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 10 client.4147.objecter ms_dispatch 0x7fe364000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:19:43.908 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T18:19:43.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe35c018dc0 con 0x7fe364146450 2024-03-25T18:19:43.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:43.909 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.905+0000 7fe3617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:43.978 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.978+0000 7fe36b214640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d98d0642-875c-461c-80aa-aa17bdd13517", "id": 5}] 2024-03-25T18:19:43.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.978+0000 7fe36b214640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d98d0642-875c-461c-80aa-aa17bdd13517", "id": 5}] 2024-03-25T18:19:43.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.978+0000 7fe36b214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:43.979 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:43.978+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d98d0642-875c-461c-80aa-aa17bdd13517", "id": 5} v 0) v1 -- 0x7fe3640b6650 con 0x7fe364146450 2024-03-25T18:19:44.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.046+0000 7fe3617fa640 1 -- 172.21.15.120:0/3166372532 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d98d0642-875c-461c-80aa-aa17bdd13517", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 736918629 0 1988412176) 0x7fe35c006550 con 0x7fe364146450 2024-03-25T18:19:44.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.046+0000 7fe3617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d98d0642-875c-461c-80aa-aa17bdd13517", "id": 5}] 2024-03-25T18:19:44.047 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.046+0000 7fe3617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:44.048 INFO:teuthology.orchestra.run.smithi120.stdout:5 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 10 librados: watch_flush enter 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 10 librados: watch_flush exit 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 10 client.4147.objecter successfully canceled tick 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 10 monclient: shutdown 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 >> v1:172.21.15.120:6789/0 conn(0x7fe364146450 legacy=0x7fe3641d49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 shutdown_connections 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.047+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 >> 172.21.15.120:0/3166372532 conn(0x7fe3640b7f40 msgr2=0x7fe364074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.048+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 shutdown_connections 2024-03-25T18:19:44.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.048+0000 7fe36b214640 1 -- 172.21.15.120:0/3166372532 wait complete. 2024-03-25T18:19:44.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.048+0000 7fe36b214640 1 librados: shutdown 2024-03-25T18:19:44.058 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 6dbbddf8-9fe5-48c5-b3fc-3bf444381159 6 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.197+0000 7fe3e0fd6640 10 monclient: get_monmap_and_config 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.197+0000 7fe3e0fd6640 10 monclient: build_initial_monmap 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.197+0000 7fe3e0fd6640 10 monclient: monmap: 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:44.198895+0000 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:44.198895+0000 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.199 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:44.200 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:44.200 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:44.200 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.198+0000 7fe3e0fd6640 1 Processor -- start 2024-03-25T18:19:44.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.198+0000 7fe3e0fd6640 1 -- start start 2024-03-25T18:19:44.200 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.198+0000 7fe3e0fd6640 10 monclient: init 2024-03-25T18:19:44.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:44.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:44.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fe3dc0b4700 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:44.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fe3dc142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:44.201 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fe3dc0a96b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:44.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc0a9d60 con 0x7fe3dc142a20 2024-03-25T18:19:44.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc13ff10 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc0aa1f0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:44.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3e0fd6640 10 monclient(hunting): authenticate will time out at 7302.355247s 2024-03-25T18:19:44.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3dad9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe3dc0a96b0 0x7fe3dc13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47466/0 (socket says 172.21.15.120:47466) 2024-03-25T18:19:44.202 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.199+0000 7fe3dad9d640 1 -- 172.21.15.120:0/634094032 learned_addr learned my addr 172.21.15.120:0/634094032 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:44.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256092121 0 0) 0x7fe3dc13ff10 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:19:44.203 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c4003520 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547844926 0 0) 0x7fe3dc0a9d60 con 0x7fe3dc142a20 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc13ff10 con 0x7fe3dc142a20 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906411039 0 0) 0x7fe3dc0aa1f0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:19:44.204 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.202+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc0a9d60 con 0x7fe3dc0b4700 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.205+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389687887 0 0) 0x7fe3dc13ff10 con 0x7fe3dc142a20 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.205+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 >> v1:172.21.15.42:6790/0 conn(0x7fe3dc0b4700 legacy=0x7fe3dc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.205+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 >> v1:172.21.15.120:6789/0 conn(0x7fe3dc0a96b0 legacy=0x7fe3dc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.205+0000 7fe3d959a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 1 monclient: found mon.noname-a 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0b4ae0 con 0x7fe3dc142a20 2024-03-25T18:19:44.207 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 10 monclient: _finish_auth 0 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe3cc001670 con 0x7fe3dc142a20 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3d959a640 10 monclient: dump: 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.208 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.209 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.209 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3e0fd6640 5 monclient: authenticate success, global_id 4152 2024-03-25T18:19:44.209 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.206+0000 7fe3e0fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3cc0017d0 con 0x7fe3dc142a20 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3d959a640 1 -- 172.21.15.120:0/634094032 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe3cc001be0 con 0x7fe3dc142a20 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3d959a640 10 monclient: dump: 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.212 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3e0fd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.207+0000 7fe3e0fd6640 4 set_mon_vals no callback set 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.208+0000 7fe3e0fd6640 10 monclient: shutdown 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.208+0000 7fe3e0fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.208+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/634094032 >> v1:172.21.15.42:6789/0 conn(0x7fe3dc142a20 legacy=0x7fe3dc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.208+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/634094032 shutdown_connections 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.208+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/634094032 >> 172.21.15.120:0/634094032 conn(0x7fe3dc0668e0 msgr2=0x7fe3dc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.208+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/634094032 shutdown_connections 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.208+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/634094032 wait complete. 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 10 monclient: build_initial_monmap 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 10 monclient: monmap: 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:44.214 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 1 librados: starting msgr at 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 1 librados: starting objecter 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 1 Processor -- start 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 1 -- start start 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 1 librados: setting wanted keys 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 1 librados: calling monclient init 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.209+0000 7fe3e0fd6640 10 monclient: init 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fe3dc0a96b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fe3dc0b4700 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fe3dc142a20 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:44.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc139100 con 0x7fe3dc0b4700 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc1393e0 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc1381a0 con 0x7fe3dc142a20 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3e0fd6640 10 monclient(hunting): authenticate will time out at 7302.366382s 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3da59c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe3dc0a96b0 0x7fe3dc137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47474/0 (socket says 172.21.15.120:47474) 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.210+0000 7fe3da59c640 1 -- 172.21.15.120:0/3378595438 learned_addr learned my addr 172.21.15.120:0/3378595438 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.211+0000 7fe3caffd640 10 client.?.objecter ms_handle_connect 0x7fe3dc0a96b0 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.211+0000 7fe3caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.211+0000 7fe3caffd640 10 client.?.objecter ms_handle_connect 0x7fe3dc0b4700 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.211+0000 7fe3caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.211+0000 7fe3caffd640 10 client.?.objecter ms_handle_connect 0x7fe3dc142a20 2024-03-25T18:19:44.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.211+0000 7fe3caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.212+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576156972 0 0) 0x7fe3dc1393e0 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.212+0000 7fe3caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.212+0000 7fe3caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.217 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.212+0000 7fe3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.212+0000 7fe3caffd640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.212+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3bc004eb0 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303683974 0 0) 0x7fe3dc139100 con 0x7fe3dc0b4700 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc1393e0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783131284 0 0) 0x7fe3dc1381a0 con 0x7fe3dc142a20 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:19:44.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.213+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc139100 con 0x7fe3dc142a20 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 852127985 0 0) 0x7fe3dc1393e0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3dc1381a0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe3cc001ae0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 10 monclient(hunting): dump: 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.220 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4151736098 0 0) 0x7fe3bc004eb0 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3dc1393e0 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe3d0001a10 con 0x7fe3dc0a96b0 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.218+0000 7fe3caffd640 10 monclient(hunting): dump: 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.221 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.219+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2284176718 0 0) 0x7fe3dc139100 con 0x7fe3dc142a20 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.219+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3bc004eb0 con 0x7fe3dc142a20 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.219+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe3d4001a50 con 0x7fe3dc142a20 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.219+0000 7fe3caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.219+0000 7fe3caffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.219+0000 7fe3caffd640 10 monclient(hunting): dump: 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.222 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.223 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1370777907 0 0) 0x7fe3dc1381a0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 >> v1:172.21.15.42:6790/0 conn(0x7fe3dc142a20 legacy=0x7fe3dc1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 >> v1:172.21.15.120:6789/0 conn(0x7fe3dc0a96b0 legacy=0x7fe3dc137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.224 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 1 monclient: found mon.a 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc139f10 con 0x7fe3dc0b4700 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 5 monclient: authenticate success, global_id 4155 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 10 monclient: _renew_subs 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc136dc0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 10 monclient: _renew_subs 2024-03-25T18:19:44.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:44.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc135510 con 0x7fe3dc0b4700 2024-03-25T18:19:44.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.221+0000 7fe3e0fd6640 1 librados: init done 2024-03-25T18:19:44.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.222+0000 7fe3e0fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.222+0000 7fe3e0fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.222+0000 7fe3e0fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:44.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.222+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc135810 con 0x7fe3dc0b4700 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3cc001210 con 0x7fe3dc0b4700 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe3cc0038a0 con 0x7fe3dc0b4700 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3caffd640 10 monclient: dump: 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.325 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.326 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.326 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.326 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.326 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.326 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe3cc003d90 con 0x7fe3dc0b4700 2024-03-25T18:19:44.326 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.324+0000 7fe3d8d99640 4 set_mon_vals no callback set 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2497778386 0 0) 0x7fe3cc005520 con 0x7fe3dc0b4700 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 10 client.4155.objecter ms_dispatch 0x7fe3dc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe3cc004f30 con 0x7fe3dc0b4700 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.327 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.325+0000 7fe3caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:44.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.396+0000 7fe3e0fd6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6dbbddf8-9fe5-48c5-b3fc-3bf444381159", "id": 6}] 2024-03-25T18:19:44.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.396+0000 7fe3e0fd6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6dbbddf8-9fe5-48c5-b3fc-3bf444381159", "id": 6}] 2024-03-25T18:19:44.397 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.396+0000 7fe3e0fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:44.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.396+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6dbbddf8-9fe5-48c5-b3fc-3bf444381159", "id": 6} v 0) v1 -- 0x7fe3dc12fd40 con 0x7fe3dc0b4700 2024-03-25T18:19:44.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.446+0000 7fe3caffd640 1 -- 172.21.15.120:0/3378595438 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6dbbddf8-9fe5-48c5-b3fc-3bf444381159", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1354088056 0 1113811849) 0x7fe3dc12fd40 con 0x7fe3dc0b4700 2024-03-25T18:19:44.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.446+0000 7fe3caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6dbbddf8-9fe5-48c5-b3fc-3bf444381159", "id": 6}] 2024-03-25T18:19:44.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.446+0000 7fe3caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:44.448 INFO:teuthology.orchestra.run.smithi120.stdout:6 2024-03-25T18:19:44.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 10 librados: watch_flush enter 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 10 librados: watch_flush exit 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 10 client.4155.objecter successfully canceled tick 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 10 monclient: shutdown 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 >> v1:172.21.15.42:6789/0 conn(0x7fe3dc0b4700 legacy=0x7fe3dc139800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 shutdown_connections 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 >> 172.21.15.120:0/3378595438 conn(0x7fe3dc0668e0 msgr2=0x7fe3dc0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 shutdown_connections 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 1 -- 172.21.15.120:0/3378595438 wait complete. 2024-03-25T18:19:44.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.447+0000 7fe3e0fd6640 1 librados: shutdown 2024-03-25T18:19:44.459 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 01b63363-95ba-4457-9474-00febf9e8452 7 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.557+0000 7fa1fb243640 10 monclient: get_monmap_and_config 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.557+0000 7fa1fb243640 10 monclient: build_initial_monmap 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.557+0000 7fa1fb243640 10 monclient: monmap: 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:19:44.558687+0000 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:19:44.558687+0000 2024-03-25T18:19:44.558 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:44.559 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.559 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:44.559 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:44.559 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:44.559 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.560 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.558+0000 7fa1fb243640 1 Processor -- start 2024-03-25T18:19:44.560 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.558+0000 7fa1fb243640 1 -- start start 2024-03-25T18:19:44.560 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.558+0000 7fa1fb243640 10 monclient: init 2024-03-25T18:19:44.560 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:44.560 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:44.560 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f40b4700 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f4142a20 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f40a96b0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f40a9d60 con 0x7fa1f40a96b0 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f413ff10 con 0x7fa1f40b4700 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f40aa1f0 con 0x7fa1f4142a20 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1fb243640 10 monclient(hunting): authenticate will time out at 7302.715083s 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1f8fdf640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa1f40b4700 0x7fa1f4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47478/0 (socket says 172.21.15.120:47478) 2024-03-25T18:19:44.561 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.559+0000 7fa1f8fdf640 1 -- 172.21.15.120:0/3223154789 learned_addr learned my addr 172.21.15.120:0/3223154789 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:44.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.561+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95077865 0 0) 0x7fa1f413ff10 con 0x7fa1f40b4700 2024-03-25T18:19:44.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.561+0000 7fa1f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.561+0000 7fa1f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.561+0000 7fa1f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.561+0000 7fa1f37fe640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:19:44.562 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.561+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e4003520 con 0x7fa1f40b4700 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.562+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963446610 0 0) 0x7fa1f40a9d60 con 0x7fa1f40a96b0 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.562+0000 7fa1f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.562+0000 7fa1f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.562+0000 7fa1f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.562+0000 7fa1f37fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.563+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f413ff10 con 0x7fa1f40a96b0 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.563+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229033824 0 0) 0x7fa1f40aa1f0 con 0x7fa1f4142a20 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.563+0000 7fa1f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.563+0000 7fa1f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.563+0000 7fa1f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.563+0000 7fa1f37fe640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:19:44.564 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.563+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f40a9d60 con 0x7fa1f4142a20 2024-03-25T18:19:44.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.564+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2410228393 0 0) 0x7fa1e4003520 con 0x7fa1f40b4700 2024-03-25T18:19:44.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.564+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 >> v1:172.21.15.42:6790/0 conn(0x7fa1f4142a20 legacy=0x7fa1f413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.564+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 >> v1:172.21.15.42:6789/0 conn(0x7fa1f40a96b0 legacy=0x7fa1f413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.565 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 1 monclient: found mon.noname-c 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40b4ae0 con 0x7fa1f40b4700 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa1e8001690 con 0x7fa1f40b4700 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1fb243640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1fb243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:44.566 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1f37fe640 10 monclient: dump: 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.567 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.565+0000 7fa1fb243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e80017f0 con 0x7fa1f40b4700 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1f37fe640 1 -- 172.21.15.120:0/3223154789 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa1e8001c00 con 0x7fa1f40b4700 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1fb243640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1fb243640 4 set_mon_vals no callback set 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.566+0000 7fa1f37fe640 10 monclient: dump: 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.569 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.567+0000 7fa1fb243640 10 monclient: shutdown 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.567+0000 7fa1fb243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.567+0000 7fa1fb243640 1 -- 172.21.15.120:0/3223154789 >> v1:172.21.15.120:6789/0 conn(0x7fa1f40b4700 legacy=0x7fa1f4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.567+0000 7fa1fb243640 1 -- 172.21.15.120:0/3223154789 shutdown_connections 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.567+0000 7fa1fb243640 1 -- 172.21.15.120:0/3223154789 >> 172.21.15.120:0/3223154789 conn(0x7fa1f40668e0 msgr2=0x7fa1f4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.567+0000 7fa1fb243640 1 -- 172.21.15.120:0/3223154789 shutdown_connections 2024-03-25T18:19:44.570 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.567+0000 7fa1fb243640 1 -- 172.21.15.120:0/3223154789 wait complete. 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 10 monclient: build_initial_monmap 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 10 monclient: monmap: 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 0 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 0.000000 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:created 0.000000 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 1 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 1 librados: starting msgr at 2024-03-25T18:19:44.571 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 1 librados: starting objecter 2024-03-25T18:19:44.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 1 Processor -- start 2024-03-25T18:19:44.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 1 -- start start 2024-03-25T18:19:44.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 1 librados: setting wanted keys 2024-03-25T18:19:44.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 1 librados: calling monclient init 2024-03-25T18:19:44.572 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.568+0000 7fa1fb243640 10 monclient: init 2024-03-25T18:19:44.574 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:44.574 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:44.574 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f40a96b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:44.574 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f40b4700 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:44.574 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f4142a20 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:44.574 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f413a960 con 0x7fa1f4142a20 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4139340 con 0x7fa1f40a96b0 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4138db0 con 0x7fa1f40b4700 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1fb243640 10 monclient(hunting): authenticate will time out at 7302.725163s 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1f8fdf640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa1f40a96b0 0x7fa1f4130f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:47494/0 (socket says 172.21.15.120:47494) 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.569+0000 7fa1f8fdf640 1 -- 172.21.15.120:0/3874893840 learned_addr learned my addr 172.21.15.120:0/3874893840 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.570+0000 7fa1f17fa640 10 client.?.objecter ms_handle_connect 0x7fa1f4142a20 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.570+0000 7fa1f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.570+0000 7fa1f17fa640 10 client.?.objecter ms_handle_connect 0x7fa1f40a96b0 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.570+0000 7fa1f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.570+0000 7fa1f17fa640 10 client.?.objecter ms_handle_connect 0x7fa1f40b4700 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.570+0000 7fa1f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.571+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461878162 0 0) 0x7fa1f4139340 con 0x7fa1f40a96b0 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.571+0000 7fa1f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.571+0000 7fa1f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.575 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.571+0000 7fa1f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.571+0000 7fa1f17fa640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.571+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8004eb0 con 0x7fa1f40a96b0 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490879907 0 0) 0x7fa1f413a960 con 0x7fa1f4142a20 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f4139340 con 0x7fa1f4142a20 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205387536 0 0) 0x7fa1f4138db0 con 0x7fa1f40b4700 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:19:44.576 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.572+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f413a960 con 0x7fa1f40b4700 2024-03-25T18:19:44.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.576+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 641651191 0 0) 0x7fa1d8004eb0 con 0x7fa1f40a96b0 2024-03-25T18:19:44.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.576+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1f4138db0 con 0x7fa1f40a96b0 2024-03-25T18:19:44.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.576+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa1e8001bc0 con 0x7fa1f40a96b0 2024-03-25T18:19:44.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.576+0000 7fa1f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.576+0000 7fa1f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:44.577 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.576+0000 7fa1f17fa640 10 monclient(hunting): dump: 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.578 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.576+0000 7fa1f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:44.580 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2170806316 0 0) 0x7fa1f4139340 con 0x7fa1f4142a20 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1d8004eb0 con 0x7fa1f4142a20 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032410367 0 0) 0x7fa1f413a960 con 0x7fa1f40b4700 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1f4139340 con 0x7fa1f40b4700 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa1ec001cb0 con 0x7fa1f4142a20 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient(hunting): dump: 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.581 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa1e0001790 con 0x7fa1f40b4700 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient(hunting): dump: 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.582 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2222733097 0 0) 0x7fa1f4138db0 con 0x7fa1f40a96b0 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 >> v1:172.21.15.42:6790/0 conn(0x7fa1f40b4700 legacy=0x7fa1f41da7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 >> v1:172.21.15.42:6789/0 conn(0x7fa1f4142a20 legacy=0x7fa1f41daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 monclient: found mon.b 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4138f40 con 0x7fa1f40a96b0 2024-03-25T18:19:44.583 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 5 monclient: authenticate success, global_id 4159 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 10 monclient: _renew_subs 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f4138750 con 0x7fa1f40a96b0 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 10 monclient: _renew_subs 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f4131640 con 0x7fa1f40a96b0 2024-03-25T18:19:44.584 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.579+0000 7fa1fb243640 1 librados: init done 2024-03-25T18:19:44.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.580+0000 7fa1fb243640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.580+0000 7fa1fb243640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.585 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.580+0000 7fa1fb243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.580+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f4139190 con 0x7fa1f40a96b0 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e8004a90 con 0x7fa1f40a96b0 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa1e8004da0 con 0x7fa1f40a96b0 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 10 monclient: dump: 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:epoch 1 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:election_strategy: 3 2024-03-25T18:19:44.586 INFO:teuthology.orchestra.run.smithi120.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa1e8005290 con 0x7fa1f40a96b0 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f2ffd640 4 set_mon_vals no callback set 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4062461993 0 0) 0x7fa1e8006a00 con 0x7fa1f40a96b0 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 10 client.4159.objecter ms_dispatch 0x7fa1f4000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:19:44.587 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.583+0000 7fa1f17fa640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-25T18:19:44.614 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:44.613+0000 7f8cc6e3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:19:44.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.675+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa1e8006450 con 0x7fa1f40a96b0 2024-03-25T18:19:44.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.675+0000 7fa1f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:44.676 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.675+0000 7fa1f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:44.684 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:44.683+0000 7f31ee442040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:19:44.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.764+0000 7fa1fb243640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "01b63363-95ba-4457-9474-00febf9e8452", "id": 7}] 2024-03-25T18:19:44.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.764+0000 7fa1fb243640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "01b63363-95ba-4457-9474-00febf9e8452", "id": 7}] 2024-03-25T18:19:44.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.764+0000 7fa1fb243640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:19:44.765 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.764+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd new", "uuid": "01b63363-95ba-4457-9474-00febf9e8452", "id": 7} v 0) v1 -- 0x7fa1f41394d0 con 0x7fa1f40a96b0 2024-03-25T18:19:44.841 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.840+0000 7fa1f17fa640 1 -- 172.21.15.120:0/3874893840 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "01b63363-95ba-4457-9474-00febf9e8452", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1247377611 0 1371665406) 0x7fa1e8006740 con 0x7fa1f40a96b0 2024-03-25T18:19:44.841 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.840+0000 7fa1f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "01b63363-95ba-4457-9474-00febf9e8452", "id": 7}] 2024-03-25T18:19:44.841 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.840+0000 7fa1f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:44.841 INFO:teuthology.orchestra.run.smithi120.stdout:7 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.841+0000 7fa1fb243640 10 librados: watch_flush enter 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 10 librados: watch_flush exit 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 10 client.4159.objecter successfully canceled tick 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 10 monclient: shutdown 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 >> v1:172.21.15.120:6789/0 conn(0x7fa1f40a96b0 legacy=0x7fa1f4130f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 shutdown_connections 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 >> 172.21.15.120:0/3874893840 conn(0x7fa1f40668e0 msgr2=0x7fa1f40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:44.843 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 shutdown_connections 2024-03-25T18:19:44.844 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 1 -- 172.21.15.120:0/3874893840 wait complete. 2024-03-25T18:19:44.844 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-25T18:19:44.842+0000 7fa1fb243640 1 librados: shutdown 2024-03-25T18:19:44.853 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.853 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:19:44.853 DEBUG:teuthology.orchestra.run.smithi042:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:19:44.856 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:19:44.856 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.856 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:19:44.856 DEBUG:teuthology.orchestra.run.smithi042:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:19:44.858 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:19:44.858 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.858 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:19:44.858 DEBUG:teuthology.orchestra.run.smithi042:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:19:44.860 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:19:44.860 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.860 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:19:44.860 DEBUG:teuthology.orchestra.run.smithi042:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:19:44.862 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:19:44.863 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.863 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:19:44.863 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:19:44.865 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:19:44.865 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.865 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:19:44.865 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:19:44.867 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:19:44.867 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.867 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:19:44.867 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:19:44.869 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:19:44.869 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:44.869 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:19:44.870 DEBUG:teuthology.orchestra.run.smithi120:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:19:44.872 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:19:44.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:44.930 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:44.929+0000 7f8cc6e3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:19:44.984 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:44.984+0000 7f31ee442040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:19:45.026 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:45.025+0000 7f8cc6e3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:19:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.037+0000 7f215d99f640 10 monclient: get_monmap_and_config 2024-03-25T18:19:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.037+0000 7f215d99f640 10 monclient: build_initial_monmap 2024-03-25T18:19:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.037+0000 7f215d99f640 10 monclient: monmap: 2024-03-25T18:19:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:19:45.038646+0000 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:19:45.038646+0000 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.038+0000 7f215d99f640 1 Processor -- start 2024-03-25T18:19:45.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.038+0000 7f215d99f640 1 -- start start 2024-03-25T18:19:45.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.038+0000 7f215d99f640 10 monclient: init 2024-03-25T18:19:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.039+0000 7f215d99f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.039+0000 7f215d99f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.039+0000 7f215d99f640 10 monclient(hunting): picked mon.noname-c con 0x7f21580b4c70 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.039+0000 7f215d99f640 10 monclient(hunting): picked mon.noname-b con 0x7f21580b4250 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.040+0000 7f215c99d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f21580b4c70 0x7f21581455f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34550/0 (socket says 172.21.15.42:34550) 2024-03-25T18:19:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.040+0000 7f215c99d640 1 -- 172.21.15.42:0/2592585394 learned_addr learned my addr 172.21.15.42:0/2592585394 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.040+0000 7f215d99f640 10 monclient(hunting): picked mon.noname-a con 0x7f21580a9dc0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.040+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580aa210 con 0x7f21580a9dc0 2024-03-25T18:19:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.040+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580ab200 con 0x7f21580b4c70 2024-03-25T18:19:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.040+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580b3d30 con 0x7f21580b4250 2024-03-25T18:19:45.042 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:45.040+0000 7f31ee442040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:19:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.041+0000 7f215d99f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.041+0000 7f215d99f640 10 monclient(hunting): authenticate will time out at 7300.267787s 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913161806 0 0) 0x7f21580ab200 con 0x7f21580b4c70 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2148003520 con 0x7f21580b4c70 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189747576 0 0) 0x7f21580b3d30 con 0x7f21580b4250 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21580ab200 con 0x7f21580b4250 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824031924 0 0) 0x7f21580aa210 con 0x7f21580a9dc0 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:19:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.046+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21580b3d30 con 0x7f21580a9dc0 2024-03-25T18:19:45.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730474128 0 0) 0x7f21580b3d30 con 0x7f21580a9dc0 2024-03-25T18:19:45.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 >> v1:172.21.15.42:6790/0 conn(0x7f21580b4250 legacy=0x7f215813f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 >> v1:172.21.15.120:6789/0 conn(0x7f21580b4c70 legacy=0x7f21581455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:45.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 1 monclient: found mon.noname-a 2024-03-25T18:19:45.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:45.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21580b2230 con 0x7f21580a9dc0 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.051+0000 7f21577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.052+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2150001860 con 0x7f21580a9dc0 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.052+0000 7f21577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.052+0000 7f21577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.052+0000 7f21577fe640 10 monclient: dump: 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:45.056 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:45.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:45.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:45.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:45.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.052+0000 7f215d99f640 5 monclient: authenticate success, global_id 4164 2024-03-25T18:19:45.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.052+0000 7f215d99f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:45.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2150001ab0 con 0x7f21580a9dc0 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f21577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f21577fe640 1 -- 172.21.15.42:0/2592585394 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2150001e40 con 0x7f21580a9dc0 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f21577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f21577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f21577fe640 10 monclient: dump: 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:45.058 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f215d99f640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.056+0000 7f215d99f640 4 set_mon_vals no callback set 2024-03-25T18:19:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.057+0000 7f215d99f640 10 monclient: shutdown 2024-03-25T18:19:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.057+0000 7f215d99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.057+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 >> v1:172.21.15.42:6789/0 conn(0x7f21580a9dc0 legacy=0x7f215814d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.058+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 shutdown_connections 2024-03-25T18:19:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.058+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 >> 172.21.15.42:0/2592585394 conn(0x7f2158066ac0 msgr2=0x7f2158066ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.058+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 shutdown_connections 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.058+0000 7f215d99f640 1 -- 172.21.15.42:0/2592585394 wait complete. 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.059+0000 7f215d99f640 10 monclient: build_initial_monmap 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.059+0000 7f215d99f640 10 monclient: monmap: 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-03-25T18:19:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-03-25T18:19:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:45.061 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.060+0000 7f215d99f640 1 librados: starting msgr at 2024-03-25T18:19:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.060+0000 7f215d99f640 1 librados: starting objecter 2024-03-25T18:19:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.060+0000 7f215d99f640 1 Processor -- start 2024-03-25T18:19:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.060+0000 7f215d99f640 1 -- start start 2024-03-25T18:19:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.060+0000 7f215d99f640 1 librados: setting wanted keys 2024-03-25T18:19:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.060+0000 7f215d99f640 1 librados: calling monclient init 2024-03-25T18:19:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.060+0000 7f215d99f640 10 monclient: init 2024-03-25T18:19:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.062+0000 7f215d99f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.062+0000 7f215d99f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.062+0000 7f215d99f640 10 monclient(hunting): picked mon.noname-a con 0x7f21580a9dc0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.063+0000 7f215c99d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f21580a9dc0 0x7f2158141870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33768/0 (socket says 172.21.15.42:33768) 2024-03-25T18:19:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.063+0000 7f215c99d640 1 -- 172.21.15.42:0/2996714525 learned_addr learned my addr 172.21.15.42:0/2996714525 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.063+0000 7f215d99f640 10 monclient(hunting): picked mon.noname-c con 0x7f21580b4250 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.063+0000 7f215d99f640 10 monclient(hunting): picked mon.noname-b con 0x7f21580b4c70 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f21557fa640 10 client.?.objecter ms_handle_connect 0x7f21580b4250 2024-03-25T18:19:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f21557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f21557fa640 10 client.?.objecter ms_handle_connect 0x7f21580a9dc0 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f21557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21581d7150 con 0x7f21580a9dc0 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21581d7fd0 con 0x7f21580b4c70 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21581d73a0 con 0x7f21580b4250 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f215d99f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.064+0000 7f215d99f640 10 monclient(hunting): authenticate will time out at 7300.291460s 2024-03-25T18:19:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.065+0000 7f21557fa640 10 client.?.objecter ms_handle_connect 0x7f21580b4c70 2024-03-25T18:19:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.065+0000 7f21557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.068+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972064054 0 0) 0x7f21581d73a0 con 0x7f21580b4250 2024-03-25T18:19:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.068+0000 7f21557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.068+0000 7f21557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.068+0000 7f21557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.068+0000 7f21557fa640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:19:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.068+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f213c004eb0 con 0x7f21580b4250 2024-03-25T18:19:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832983594 0 0) 0x7f21581d7fd0 con 0x7f21580b4c70 2024-03-25T18:19:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:19:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21581d73a0 con 0x7f21580b4c70 2024-03-25T18:19:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303932312 0 0) 0x7f21581d7150 con 0x7f21580a9dc0 2024-03-25T18:19:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:19:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.073+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21581d7fd0 con 0x7f21580a9dc0 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.075+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747449572 0 0) 0x7f213c004eb0 con 0x7f21580b4250 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.075+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21581d7150 con 0x7f21580b4250 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.075+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2144001a50 con 0x7f21580b4250 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.075+0000 7f21557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.075+0000 7f21557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.075+0000 7f21557fa640 10 monclient(hunting): dump: 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.075+0000 7f21557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:45.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2383174339 0 0) 0x7f21581d7150 con 0x7f21580b4250 2024-03-25T18:19:45.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 >> v1:172.21.15.120:6789/0 conn(0x7f21580b4c70 legacy=0x7f21581d78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 >> v1:172.21.15.42:6789/0 conn(0x7f21580a9dc0 legacy=0x7f2158141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 1 monclient: found mon.c 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158143870 con 0x7f21580b4250 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f21557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f215d99f640 5 monclient: authenticate success, global_id 4166 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f215d99f640 10 monclient: _renew_subs 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f215d99f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2158143ac0 con 0x7f21580b4250 2024-03-25T18:19:45.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f215d99f640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f215d99f640 10 monclient: _renew_subs 2024-03-25T18:19:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.081+0000 7f215d99f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:19:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.082+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21580b1130 con 0x7f21580b4250 2024-03-25T18:19:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.082+0000 7f215d99f640 1 librados: init done 2024-03-25T18:19:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.083+0000 7f215d99f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.083+0000 7f215d99f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.083+0000 7f215d99f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:19:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.083+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21581d7530 con 0x7f21580b4250 2024-03-25T18:19:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.122+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21440035f0 con 0x7f21580b4250 2024-03-25T18:19:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.122+0000 7f21557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.122+0000 7f2156ffd640 4 set_mon_vals no callback set 2024-03-25T18:19:45.163 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:45.162+0000 7f8cc6e3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:19:45.169 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-25T18:19:45.168+0000 7f31ee442040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2144003900 con 0x7f21580b4250 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 10 monclient: dump: 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.446 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2144003df0 con 0x7f21580b4250 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f21440057a0 con 0x7f21580b4250 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 10 client.4166.objecter ms_dispatch 0x7f2158000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 20 client.4166.objecter dump_active .. 0 homeless 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2144005bc0 con 0x7f21580b4250 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.444+0000 7f21557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.569+0000 7f215d99f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.569+0000 7f215d99f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.569+0000 7f215d99f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:19:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.569+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21580627b0 con 0x7f21580b4250 2024-03-25T18:19:45.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.768+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3240790519 0 0) 0x7f21440050d0 con 0x7f21580b4250 2024-03-25T18:19:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.888+0000 7f21557fa640 1 -- 172.21.15.42:0/2996714525 <== mon.2 v1:172.21.15.42:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f2144014a20 con 0x7f21580b4250 2024-03-25T18:19:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.888+0000 7f21557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.888+0000 7f21557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:45.890 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-25T18:19:45.890 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 10 librados: watch_flush enter 2024-03-25T18:19:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 10 librados: watch_flush exit 2024-03-25T18:19:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 20 client.4166.objecter shutdown clearing up homeless session... 2024-03-25T18:19:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 10 client.4166.objecter successfully canceled tick 2024-03-25T18:19:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 10 monclient: shutdown 2024-03-25T18:19:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 >> v1:172.21.15.42:6790/0 conn(0x7f21580b4250 legacy=0x7f2158140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 shutdown_connections 2024-03-25T18:19:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 >> 172.21.15.42:0/2996714525 conn(0x7f2158066ac0 msgr2=0x7f215813a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 shutdown_connections 2024-03-25T18:19:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.890+0000 7f215d99f640 1 -- 172.21.15.42:0/2996714525 wait complete. 2024-03-25T18:19:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-25T18:19:45.891+0000 7f215d99f640 1 librados: shutdown 2024-03-25T18:19:45.908 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:19:45.908 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:19:46.714 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:46.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.858+0000 7fc7ae8fb640 10 monclient: get_monmap_and_config 2024-03-25T18:19:46.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.858+0000 7fc7ae8fb640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.858+0000 7fc7ae8fb640 10 monclient: monmap: 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:19:46.859737+0000 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:19:46.859737+0000 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:46.860 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:46.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.859+0000 7fc7ae8fb640 1 Processor -- start 2024-03-25T18:19:46.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.859+0000 7fc7ae8fb640 1 -- start start 2024-03-25T18:19:46.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.859+0000 7fc7ae8fb640 10 monclient: init 2024-03-25T18:19:46.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:46.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a00389e0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:46.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a0038dc0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:46.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a0030660 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:46.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a00c54c0 con 0x7fc7a0038dc0 2024-03-25T18:19:46.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a0036430 con 0x7fc7a00389e0 2024-03-25T18:19:46.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a0039840 con 0x7fc7a0030660 2024-03-25T18:19:46.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.860+0000 7fc7ae8fb640 10 monclient(hunting): authenticate will time out at 7302.087508s 2024-03-25T18:19:46.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.887+0000 7fc7ad8f9640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc7a00389e0 0x7fc7a0033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34656/0 (socket says 172.21.15.42:34656) 2024-03-25T18:19:46.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.887+0000 7fc7ad8f9640 1 -- 172.21.15.42:0/96007426 learned_addr learned my addr 172.21.15.42:0/96007426 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:46.951 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:46.950+0000 7f8c67b11640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:19:46.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.960+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413083304 0 0) 0x7fc7a0036430 con 0x7fc7a00389e0 2024-03-25T18:19:46.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.960+0000 7fc7ac8f7640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.960+0000 7fc7ac8f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.960+0000 7fc7ac8f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.960+0000 7fc7ac8f7640 10 monclient(hunting): my global_id is 4183 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.960+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc794003520 con 0x7fc7a00389e0 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.966+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2224768464 0 0) 0x7fc794003520 con 0x7fc7a00389e0 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.966+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 >> v1:172.21.15.42:6790/0 conn(0x7fc7a0030660 legacy=0x7fc7a00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.966+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 >> v1:172.21.15.42:6789/0 conn(0x7fc7a0038dc0 legacy=0x7fc7a00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.966+0000 7fc7ac8f7640 10 monclient: _finish_hunting 0 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.966+0000 7fc7ac8f7640 1 monclient: found mon.noname-c 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.966+0000 7fc7ac8f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0035690 con 0x7fc7a00389e0 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 10 monclient: _finish_auth 0 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 10 monclient: _check_auth_tickets 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc7a4001690 con 0x7fc7a00389e0 2024-03-25T18:19:46.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ac8f7640 10 monclient: dump: 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ae8fb640 5 monclient: authenticate success, global_id 4183 2024-03-25T18:19:46.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.967+0000 7fc7ae8fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:46.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7a4001860 con 0x7fc7a00389e0 2024-03-25T18:19:46.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ac8f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:46.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ac8f7640 1 -- 172.21.15.42:0/96007426 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc7a4001cb0 con 0x7fc7a00389e0 2024-03-25T18:19:46.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ac8f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ac8f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ac8f7640 10 monclient: dump: 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:46.971 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ae8fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.969+0000 7fc7ae8fb640 4 set_mon_vals no callback set 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.970+0000 7fc7ae8fb640 10 monclient: shutdown 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.970+0000 7fc7ae8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.970+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/96007426 >> v1:172.21.15.120:6789/0 conn(0x7fc7a00389e0 legacy=0x7fc7a0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.970+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/96007426 shutdown_connections 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.970+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/96007426 >> 172.21.15.42:0/96007426 conn(0x7fc7a0005c00 msgr2=0x7fc7a0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.971+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/96007426 shutdown_connections 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.971+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/96007426 wait complete. 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 10 monclient: monmap: 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 1 librados: starting msgr at 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 1 librados: starting objecter 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 1 Processor -- start 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 1 -- start start 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 1 librados: setting wanted keys 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 1 librados: calling monclient init 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.972+0000 7fc7ae8fb640 10 monclient: init 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.973+0000 7fc7ae8fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.973+0000 7fc7ae8fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.973+0000 7fc7ae8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a0030660 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.973+0000 7fc7ae8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a00389e0 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:46.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.974+0000 7fc7ae8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a00c77f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:46.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.978+0000 7fc7ae0fa640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc7a00c77f0 0x7fc7a00c1f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34660/0 (socket says 172.21.15.42:34660) 2024-03-25T18:19:46.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.978+0000 7fc7ae0fa640 1 -- 172.21.15.42:0/644761552 learned_addr learned my addr 172.21.15.42:0/644761552 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:46.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.978+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a000a2b0 con 0x7fc7a00389e0 2024-03-25T18:19:46.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.978+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a002f1e0 con 0x7fc7a00c77f0 2024-03-25T18:19:46.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.978+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a00c2690 con 0x7fc7a0030660 2024-03-25T18:19:46.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.978+0000 7fc7ae8fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.978+0000 7fc7ae8fb640 10 monclient(hunting): authenticate will time out at 7302.204867s 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.982+0000 7fc79e7fc640 10 client.?.objecter ms_handle_connect 0x7fc7a00c77f0 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.982+0000 7fc79e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.985+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132294209 0 0) 0x7fc7a002f1e0 con 0x7fc7a00c77f0 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.985+0000 7fc79e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.985+0000 7fc79e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.985+0000 7fc79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.985+0000 7fc79e7fc640 10 monclient(hunting): my global_id is 4186 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.985+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c004eb0 con 0x7fc7a00c77f0 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.993+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1054015029 0 0) 0x7fc78c004eb0 con 0x7fc7a00c77f0 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.993+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7a002f1e0 con 0x7fc7a00c77f0 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.993+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc7a8069d20 con 0x7fc7a00c77f0 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.993+0000 7fc79e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.993+0000 7fc79e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.993+0000 7fc79e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:46.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:46.993+0000 7fc79e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:47.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.004+0000 7fc79e7fc640 10 client.?.objecter ms_handle_connect 0x7fc7a0030660 2024-03-25T18:19:47.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.004+0000 7fc79e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.011+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183972415 0 0) 0x7fc7a00c2690 con 0x7fc7a0030660 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.011+0000 7fc79e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.011+0000 7fc79e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.011+0000 7fc79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.011+0000 7fc79e7fc640 10 monclient(hunting): my global_id is 4178 2024-03-25T18:19:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.011+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c004eb0 con 0x7fc7a0030660 2024-03-25T18:19:47.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.029+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3152058354 0 0) 0x7fc78c004eb0 con 0x7fc7a0030660 2024-03-25T18:19:47.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.029+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7a00c2690 con 0x7fc7a0030660 2024-03-25T18:19:47.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.030+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc7a4001b70 con 0x7fc7a0030660 2024-03-25T18:19:47.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.030+0000 7fc79e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.030+0000 7fc79e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.030+0000 7fc79e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:47.031 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:47.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.061+0000 7fc79e7fc640 10 client.?.objecter ms_handle_connect 0x7fc7a00389e0 2024-03-25T18:19:47.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.061+0000 7fc79e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.064+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163132596 0 0) 0x7fc7a000a2b0 con 0x7fc7a00389e0 2024-03-25T18:19:47.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.064+0000 7fc79e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.064+0000 7fc79e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.064+0000 7fc79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.064+0000 7fc79e7fc640 10 monclient(hunting): my global_id is 4182 2024-03-25T18:19:47.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.064+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c004eb0 con 0x7fc7a00389e0 2024-03-25T18:19:47.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.072+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742404521 0 0) 0x7fc78c004eb0 con 0x7fc7a00389e0 2024-03-25T18:19:47.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.072+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7a000a2b0 con 0x7fc7a00389e0 2024-03-25T18:19:47.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.072+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc798001790 con 0x7fc7a00389e0 2024-03-25T18:19:47.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.072+0000 7fc79e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.072+0000 7fc79e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.072+0000 7fc79e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:47.074 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:47.074 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:19:47.073+0000 7f8c67b11640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4012369917 0 0) 0x7fc7a000a2b0 con 0x7fc7a00389e0 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 >> v1:172.21.15.42:6790/0 conn(0x7fc7a0030660 legacy=0x7fc7a0164150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 >> v1:172.21.15.120:6789/0 conn(0x7fc7a00c77f0 legacy=0x7fc7a00c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 1 monclient: found mon.a 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a00c2820 con 0x7fc7a00389e0 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc79e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.078+0000 7fc7ae8fb640 5 monclient: authenticate success, global_id 4182 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 10 monclient: _renew_subs 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a002f700 con 0x7fc7a00389e0 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 10 client.4182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 10 monclient: _renew_subs 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a002fb90 con 0x7fc7a00389e0 2024-03-25T18:19:47.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.079+0000 7fc7ae8fb640 1 librados: init done 2024-03-25T18:19:47.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.080+0000 7fc7ae8fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.080+0000 7fc7ae8fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.080+0000 7fc7ae8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:47.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.080+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a0007b90 con 0x7fc7a00389e0 2024-03-25T18:19:47.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.109+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc798001c80 con 0x7fc7a00389e0 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.109+0000 7fc79e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.109+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc7980037b0 con 0x7fc7a00389e0 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.109+0000 7fc79e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.109+0000 7fc79e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.109+0000 7fc79e7fc640 10 monclient: dump: 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:47.110 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:47.111 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:47.111 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:47.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.109+0000 7fc79ffff640 4 set_mon_vals no callback set 2024-03-25T18:19:47.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.110+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2259739221 0 0) 0x7fc7980039d0 con 0x7fc7a00389e0 2024-03-25T18:19:47.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.111+0000 7fc7ad8f9640 10 monclient: get_auth_request con 0x7fc78c06deb0 auth_method 0 2024-03-25T18:19:47.112 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.111+0000 7fc79e7fc640 10 client.4182.objecter ms_handle_connect 0x7fc78c06deb0 2024-03-25T18:19:47.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.114+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fc798053370 con 0x7fc7a00389e0 2024-03-25T18:19:47.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.114+0000 7fc79e7fc640 10 client.4182.objecter ms_dispatch 0x7fc7a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:47.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.114+0000 7fc79e7fc640 3 client.4182.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:47.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.114+0000 7fc79e7fc640 3 client.4182.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:47.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.114+0000 7fc79e7fc640 20 client.4182.objecter dump_active .. 0 homeless 2024-03-25T18:19:47.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.524+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc79804ee80 con 0x7fc7a00389e0 2024-03-25T18:19:47.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.524+0000 7fc79e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.524+0000 7fc79e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:47.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.897+0000 7fc7ae8fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.897+0000 7fc7ae8fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.897+0000 7fc7ae8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:47.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:47.897+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7a002f370 con 0x7fc7a00389e0 2024-03-25T18:19:48.019 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.018+0000 7fc79e7fc640 1 -- 172.21.15.42:0/644761552 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fc798050db0 con 0x7fc7a00389e0 2024-03-25T18:19:48.019 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.018+0000 7fc79e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:48.019 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.018+0000 7fc79e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:48.019 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:19:48.019 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.020+0000 7fc7ae8fb640 10 librados: watch_flush enter 2024-03-25T18:19:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 10 librados: watch_flush exit 2024-03-25T18:19:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 20 client.4182.objecter shutdown clearing up homeless session... 2024-03-25T18:19:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 10 client.4182.objecter successfully canceled tick 2024-03-25T18:19:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 >> v1:172.21.15.42:6800/3081750888 conn(0x7fc78c06deb0 legacy=0x7fc78c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 10 monclient: shutdown 2024-03-25T18:19:48.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:48.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 >> v1:172.21.15.42:6789/0 conn(0x7fc7a00389e0 legacy=0x7fc7a002e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 shutdown_connections 2024-03-25T18:19:48.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 >> 172.21.15.42:0/644761552 conn(0x7fc7a0005c00 msgr2=0x7fc7a0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:48.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 shutdown_connections 2024-03-25T18:19:48.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 1 -- 172.21.15.42:0/644761552 wait complete. 2024-03-25T18:19:48.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:48.021+0000 7fc7ae8fb640 1 librados: shutdown 2024-03-25T18:19:48.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:51.613 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:06.682 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:51.695 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:06.764 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:51.697 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:06.763 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:51.744 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:06.800 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:51.760 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:06.827 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:51.821 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:06.877 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.176 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:07.221 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.236 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:07.281 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.242 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-25T18:19:52.235+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:52.309 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-25T18:19:52.303+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:52.377 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-25T18:19:52.371+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:52.418 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:07.469 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.458 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:07.514 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.481 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:07.533 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.520 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:07.576 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.532 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:07.605 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.593 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:07.666 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.636 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:07.710 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.694 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:07.763 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.696 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:07.769 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.775 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:07.842 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.785 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-25T18:19:52.779+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:52.832 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:07.888 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.026 INFO:tasks.ceph.osd.3.smithi042.stderr:2024-03-25T18:19:53.019+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:53.059 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:08.129 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.078 INFO:tasks.ceph.osd.1.smithi042.stderr:2024-03-25T18:19:53.072+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:53.140 INFO:tasks.ceph.osd.0.smithi042.stderr:2024-03-25T18:19:53.134+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:53.164 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:08.231 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.239 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:08.295 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.241 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:08.287 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.296 INFO:tasks.ceph.osd.2.smithi042.stderr:2024-03-25T18:19:53.287+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:53.413 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:08.482 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.500 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:08.552 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.512 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:08.579 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.585 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:08.641 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.612 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:08.658 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.615 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:08.672 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.641 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:08.714 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.703 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:08.773 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.780 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:08.846 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.801 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:08.874 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.869 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:08.921 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.908 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:08.965 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.965 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:09.020 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:53.977 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:09.023 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.000 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:09.070 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.012 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:09.085 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.044 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:09.110 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.176 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:09.249 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.181 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:09.233 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.196 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:09.253 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.270 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:09.316 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.276 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:09.345 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.313 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:09.379 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.325 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:09.380 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.373 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:09.446 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.447 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:09.499 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.494 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:09.550 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.543 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:09.616 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.566 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:09.611 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.573 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:09.643 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.591 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:09.657 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.593 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:09.648 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.662 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:09.735 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.711 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:09.763 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.781 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:09.838 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.837 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:09.910 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.841 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:54.864 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:09.919 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.864 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:09.910 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.872 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:09.938 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.881 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:09.950 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.930+0000 7f6dc1a50640 10 monclient: get_monmap_and_config 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.930+0000 7f6dc1a50640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.930+0000 7f6dc1a50640 10 monclient: monmap: 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:19:54.931328+0000 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:19:54.931328+0000 2024-03-25T18:19:54.931 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.932 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:54.932 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:54.932 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:19:54.932 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:19:54.932 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:54.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.930+0000 7f6dc1a50640 1 Processor -- start 2024-03-25T18:19:54.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.930+0000 7f6dc1a50640 1 -- start start 2024-03-25T18:19:54.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.930+0000 7f6dc1a50640 10 monclient: init 2024-03-25T18:19:54.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 10 monclient(hunting): picked mon.noname-b con 0x7f6dbc0b10d0 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 10 monclient(hunting): picked mon.noname-c con 0x7f6dbc0b4360 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 10 monclient(hunting): picked mon.noname-a con 0x7f6dbc0a9470 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc0a9ab0 con 0x7f6dbc0a9470 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc0b2400 con 0x7f6dbc0b4360 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc0aa2a0 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.931+0000 7f6dc1a50640 10 monclient(hunting): authenticate will time out at 7310.158127s 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.932+0000 7f6dba7fc640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f6dbc0b4360 0x7f6dbc0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:43432/0 (socket says 172.21.15.42:43432) 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.932+0000 7f6dbaffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f6dbc0b10d0 0x7f6dbc0b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:55956/0 (socket says 172.21.15.42:55956) 2024-03-25T18:19:54.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.932+0000 7f6dba7fc640 1 -- 172.21.15.42:0/24303963 learned_addr learned my addr 172.21.15.42:0/24303963 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:54.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864671478 0 0) 0x7f6dbc0b2400 con 0x7f6dbc0b4360 2024-03-25T18:19:54.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:19:54.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da8003520 con 0x7f6dbc0b4360 2024-03-25T18:19:54.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556119490 0 0) 0x7f6dbc0aa2a0 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc0b2400 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285506004 0 0) 0x7f6dbc0a9ab0 con 0x7f6dbc0a9470 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.934+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc0aa2a0 con 0x7f6dbc0a9470 2024-03-25T18:19:54.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3527801438 0 0) 0x7f6da8003520 con 0x7f6dbc0b4360 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 >> v1:172.21.15.42:6790/0 conn(0x7f6dbc0b10d0 legacy=0x7f6dbc0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 >> v1:172.21.15.42:6789/0 conn(0x7f6dbc0a9470 legacy=0x7f6dbc14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 1 monclient: found mon.noname-c 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc0b14b0 con 0x7f6dbc0b4360 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6da4001690 con 0x7f6dbc0b4360 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6dc1a50640 5 monclient: authenticate success, global_id 4213 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6dc1a50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6db9ffb640 10 monclient: dump: 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.943 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.937+0000 7f6dc1a50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6da40017f0 con 0x7f6dbc0b4360 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6db9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6db9ffb640 1 -- 172.21.15.42:0/24303963 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6da4001c00 con 0x7f6dbc0b4360 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6db9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6db9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6db9ffb640 10 monclient: dump: 2024-03-25T18:19:54.944 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 4 set_mon_vals no callback set 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 10 monclient: shutdown 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:54.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 1 -- 172.21.15.42:0/24303963 >> v1:172.21.15.120:6789/0 conn(0x7f6dbc0b4360 legacy=0x7f6dbc0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 1 -- 172.21.15.42:0/24303963 shutdown_connections 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 1 -- 172.21.15.42:0/24303963 >> 172.21.15.42:0/24303963 conn(0x7f6dbc0617a0 msgr2=0x7f6dbc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.939+0000 7f6dc1a50640 1 -- 172.21.15.42:0/24303963 shutdown_connections 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.940+0000 7f6dc1a50640 1 -- 172.21.15.42:0/24303963 wait complete. 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.940+0000 7f6dc1a50640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.940+0000 7f6dc1a50640 10 monclient: monmap: 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:19:54.946 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.940+0000 7f6dc1a50640 1 librados: starting msgr at 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.940+0000 7f6dc1a50640 1 librados: starting objecter 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.940+0000 7f6dc1a50640 1 Processor -- start 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 1 -- start start 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 1 librados: setting wanted keys 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 1 librados: calling monclient init 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient: init 2024-03-25T18:19:54.951 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.951 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient(hunting): picked mon.noname-c con 0x7f6dbc0a9470 addr v1:172.21.15.42:6790/0 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient(hunting): picked mon.noname-b con 0x7f6dbc0b10d0 addr v1:172.21.15.120:6789/0 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient(hunting): picked mon.noname-a con 0x7f6dbc0b4360 addr v1:172.21.15.42:6789/0 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc0ac950 con 0x7f6dbc0b4360 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc1db640 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc1db310 con 0x7f6dbc0a9470 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.941+0000 7f6dc1a50640 10 monclient(hunting): authenticate will time out at 7310.168252s 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6dbb7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6dbc0b4360 0x7f6dbc13e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46286/0 (socket says 172.21.15.42:46286) 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6dbb7fe640 1 -- 172.21.15.42:0/62436267 learned_addr learned my addr 172.21.15.42:0/62436267 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6db3fff640 10 client.?.objecter ms_handle_connect 0x7f6dbc0b10d0 2024-03-25T18:19:54.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6db3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6db3fff640 10 client.?.objecter ms_handle_connect 0x7f6dbc0b4360 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6db3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6db3fff640 10 client.?.objecter ms_handle_connect 0x7f6dbc0a9470 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.942+0000 7f6db3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294933820 0 0) 0x7f6dbc0ac950 con 0x7f6dbc0b4360 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c004eb0 con 0x7f6dbc0b4360 2024-03-25T18:19:54.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171566020 0 0) 0x7f6dbc1db640 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc0ac950 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197533298 0 0) 0x7f6dbc1db310 con 0x7f6dbc0a9470 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.944+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc1db640 con 0x7f6dbc0a9470 2024-03-25T18:19:54.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492655133 0 0) 0x7f6d9c004eb0 con 0x7f6dbc0b4360 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dbc1db310 con 0x7f6dbc0b4360 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6db4001c70 con 0x7f6dbc0b4360 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 10 monclient(hunting): dump: 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:54.955 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:54.956 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:54.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:54.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 183303227 0 0) 0x7f6dbc0ac950 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9c004eb0 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6da4001b00 con 0x7f6dbc0b10d0 2024-03-25T18:19:54.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.949+0000 7f6db3fff640 10 monclient(hunting): dump: 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.950+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1623072843 0 0) 0x7f6dbc1db640 con 0x7f6dbc0a9470 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.950+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dbc0ac950 con 0x7f6dbc0a9470 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.950+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6dac001930 con 0x7f6dbc0a9470 2024-03-25T18:19:54.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.950+0000 7f6db3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.950+0000 7f6db3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:54.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.950+0000 7f6db3fff640 10 monclient(hunting): dump: 2024-03-25T18:19:54.960 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:54.960 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:54.960 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:54.962 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:54.962 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:54.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2003410180 0 0) 0x7f6dbc1db310 con 0x7f6dbc0b4360 2024-03-25T18:19:54.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 >> v1:172.21.15.42:6790/0 conn(0x7f6dbc0a9470 legacy=0x7f6dbc1dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 >> v1:172.21.15.120:6789/0 conn(0x7f6dbc0b10d0 legacy=0x7f6dbc0af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 1 monclient: found mon.a 2024-03-25T18:19:54.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:54.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc0afe60 con 0x7f6dbc0b4360 2024-03-25T18:19:54.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6db3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 10 monclient: _renew_subs 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc0b00b0 con 0x7f6dbc0b4360 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 10 monclient: _renew_subs 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc0afa40 con 0x7f6dbc0b4360 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.952+0000 7f6dc1a50640 1 librados: init done 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.954+0000 7f6dc1a50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.954+0000 7f6dc1a50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.954+0000 7f6dc1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:54.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.954+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dbc1db4a0 con 0x7f6dbc0b4360 2024-03-25T18:19:54.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.955+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6db4003990 con 0x7f6dbc0b4360 2024-03-25T18:19:54.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.955+0000 7f6db3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:54.955+0000 7f6db97fa640 4 set_mon_vals no callback set 2024-03-25T18:19:54.970 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:10.042 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:54.983 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:10.035 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.069 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:10.125 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.129 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:10.203 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.136 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:10.202 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.155 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:10.210 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.166 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:10.212 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.194 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:10.264 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.211+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6db4003c00 con 0x7f6dbc0b4360 2024-03-25T18:19:55.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.211+0000 7f6db3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.211+0000 7f6db3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.211+0000 7f6db3fff640 10 monclient: dump: 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:19:55.212 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:19:55.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.211+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f6db4003de0 con 0x7f6dbc0b4360 2024-03-25T18:19:55.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.212+0000 7f6dbaffd640 10 monclient: get_auth_request con 0x7f6d9c06e0f0 auth_method 0 2024-03-25T18:19:55.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.212+0000 7f6db3fff640 10 client.4212.objecter ms_handle_connect 0x7f6d9c06e0f0 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f6db40047e0 con 0x7f6dbc0b4360 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 10 client.4212.objecter ms_dispatch 0x7f6dbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 20 client.4212.objecter dump_active .. 0 homeless 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6db40504c0 con 0x7f6dbc0b4360 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:55.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.217+0000 7f6db3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:55.249 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:10.301 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.271 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:10.344 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.360 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:10.416 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.398+0000 7f6dc1a50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.398+0000 7f6dc1a50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.398+0000 7f6dc1a50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:19:55.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.398+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dbc1db7d0 con 0x7f6dbc0b4360 2024-03-25T18:19:55.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.413+0000 7f6db3fff640 1 -- 172.21.15.42:0/62436267 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f6db40523f0 con 0x7f6dbc0b4360 2024-03-25T18:19:55.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.413+0000 7f6db3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.413+0000 7f6db3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:55.414 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:19:55.414 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:55.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 10 librados: watch_flush enter 2024-03-25T18:19:55.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 10 librados: watch_flush exit 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 20 client.4212.objecter shutdown clearing up homeless session... 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 10 client.4212.objecter successfully canceled tick 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 >> v1:172.21.15.42:6800/3081750888 conn(0x7f6d9c06e0f0 legacy=0x7f6d9c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 10 monclient: shutdown 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 >> v1:172.21.15.42:6789/0 conn(0x7f6dbc0b4360 legacy=0x7f6dbc13e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 shutdown_connections 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 >> 172.21.15.42:0/62436267 conn(0x7f6dbc0617a0 msgr2=0x7f6dbc145e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 shutdown_connections 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 1 -- 172.21.15.42:0/62436267 wait complete. 2024-03-25T18:19:55.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:19:55.415+0000 7f6dc1a50640 1 librados: shutdown 2024-03-25T18:19:55.425 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:55.426 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:10.499 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.456 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:10.512 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.462 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:10.508 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.509 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:10.564 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.522 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:10.574 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.559 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:10.633 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.611 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:10.677 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.665 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:10.731 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.715 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:10.788 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.719 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:10.789 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.760 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:10.806 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.784 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:10.854 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.794 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:10.846 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.803 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:10.859 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.851 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:10.925 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.897 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:10.953 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:55.952 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:11.008 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.004 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:11.077 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.148 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:11.221 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.287 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:11.339 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.301 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:11.374 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.335 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:11.391 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.344 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:11.396 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.372 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:11.418 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.391 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:11.446 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.430 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:11.476 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.703 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:11.777 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.779 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:11.852 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.834 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:11.908 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:56.904 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:11.977 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:01.397 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:16.463 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:01.716 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:16.772 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:01.775 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:16.844 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:02.100 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:17.152 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:02.231 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:02.261 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:17.316 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:02.322 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.321+0000 7f230bc93640 10 monclient: get_monmap_and_config 2024-03-25T18:20:02.322 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.321+0000 7f230bc93640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.322 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.321+0000 7f230bc93640 10 monclient: monmap: 2024-03-25T18:20:02.322 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:02.322 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:02.322 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:02.322633+0000 2024-03-25T18:20:02.322 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:02.322633+0000 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.322+0000 7f230bc93640 1 Processor -- start 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.322+0000 7f230bc93640 1 -- start start 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.322+0000 7f230bc93640 10 monclient: init 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.322+0000 7f230bc93640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.322+0000 7f230bc93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.322+0000 7f230bc93640 10 monclient(hunting): picked mon.noname-a con 0x7f23040b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.322+0000 7f230bc93640 10 monclient(hunting): picked mon.noname-c con 0x7f23040b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230bc93640 10 monclient(hunting): picked mon.noname-b con 0x7f230414be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230bc93640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040af940 con 0x7f23040b4d60 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230bc93640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040b0a70 con 0x7f23040b1730 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230bc93640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040b0d50 con 0x7f230414be90 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230bc93640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230bc93640 10 monclient(hunting): authenticate will time out at 7317.549777s 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230922e640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f23040b1730 0x7f23040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:46772/0 (socket says 172.21.15.42:46772) 2024-03-25T18:20:02.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.323+0000 7f230922e640 1 -- 172.21.15.42:0/2022383748 learned_addr learned my addr 172.21.15.42:0/2022383748 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:02.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.325+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206190072 0 0) 0x7f23040af940 con 0x7f23040b4d60 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.325+0000 7f2308a2d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.325+0000 7f2308a2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.325+0000 7f2308a2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.325+0000 7f2308a2d640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.325+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0003520 con 0x7f23040b4d60 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.326+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335884950 0 0) 0x7f23040b0d50 con 0x7f230414be90 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.326+0000 7f2308a2d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.326+0000 7f2308a2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.326+0000 7f2308a2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.326+0000 7f2308a2d640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.326+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0003840 con 0x7f230414be90 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.327+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846112975 0 0) 0x7f23040b0a70 con 0x7f23040b1730 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.327+0000 7f2308a2d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.327+0000 7f2308a2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.327+0000 7f2308a2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.327+0000 7f2308a2d640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.327+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23040b0d50 con 0x7f23040b1730 2024-03-25T18:20:02.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671855187 0 0) 0x7f22f0003520 con 0x7f23040b4d60 2024-03-25T18:20:02.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 >> v1:172.21.15.42:6790/0 conn(0x7f230414be90 legacy=0x7f230414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 >> v1:172.21.15.120:6789/0 conn(0x7f23040b1730 legacy=0x7f23040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 1 monclient: found mon.noname-a 2024-03-25T18:20:02.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:20:02.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23040b1030 con 0x7f23040b4d60 2024-03-25T18:20:02.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f2308a2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f230bc93640 5 monclient: authenticate success, global_id 4215 2024-03-25T18:20:02.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.330+0000 7f230bc93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:02.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2300001560 con 0x7f23040b4d60 2024-03-25T18:20:02.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:02.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 10 monclient: dump: 2024-03-25T18:20:02.334 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f230bc93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23000017f0 con 0x7f23040b4d60 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f230bc93640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f230bc93640 4 set_mon_vals no callback set 2024-03-25T18:20:02.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 1 -- 172.21.15.42:0/2022383748 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2300001560 con 0x7f23040b4d60 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.332+0000 7f2308a2d640 10 monclient: dump: 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.333+0000 7f230bc93640 10 monclient: shutdown 2024-03-25T18:20:02.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.333+0000 7f230bc93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.333+0000 7f230bc93640 1 -- 172.21.15.42:0/2022383748 >> v1:172.21.15.42:6789/0 conn(0x7f23040b4d60 legacy=0x7f230413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.333+0000 7f230bc93640 1 -- 172.21.15.42:0/2022383748 shutdown_connections 2024-03-25T18:20:02.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.333+0000 7f230bc93640 1 -- 172.21.15.42:0/2022383748 >> 172.21.15.42:0/2022383748 conn(0x7f23040617a0 msgr2=0x7f2304061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.333+0000 7f230bc93640 1 -- 172.21.15.42:0/2022383748 shutdown_connections 2024-03-25T18:20:02.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.333+0000 7f230bc93640 1 -- 172.21.15.42:0/2022383748 wait complete. 2024-03-25T18:20:02.338 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 10 monclient: monmap: 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 1 librados: starting msgr at 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 1 librados: starting objecter 2024-03-25T18:20:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 1 Processor -- start 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 1 -- start start 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 1 librados: setting wanted keys 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 1 librados: calling monclient init 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.334+0000 7f230bc93640 10 monclient: init 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 10 monclient(hunting): picked mon.noname-a con 0x7f23040b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 10 monclient(hunting): picked mon.noname-c con 0x7f23040b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 10 monclient(hunting): picked mon.noname-b con 0x7f230414be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23041395e0 con 0x7f23040b1730 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2304134f90 con 0x7f230414be90 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23041433b0 con 0x7f23040b4d60 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.335+0000 7f230bc93640 10 monclient(hunting): authenticate will time out at 7317.562085s 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f230922e640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f23040b4d60 0x7f2304134170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:49256/0 (socket says 172.21.15.42:49256) 2024-03-25T18:20:02.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f230922e640 1 -- 172.21.15.42:0/2524279107 learned_addr learned my addr 172.21.15.42:0/2524279107 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:02.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f22fa7fc640 10 client.?.objecter ms_handle_connect 0x7f23040b4d60 2024-03-25T18:20:02.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f22fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f22fa7fc640 10 client.?.objecter ms_handle_connect 0x7f23040b1730 2024-03-25T18:20:02.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f22fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f22fa7fc640 10 client.?.objecter ms_handle_connect 0x7f230414be90 2024-03-25T18:20:02.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.336+0000 7f22fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.338+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479261949 0 0) 0x7f23041433b0 con 0x7f23040b4d60 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.338+0000 7f22fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e8004eb0 con 0x7f23040b4d60 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772384941 0 0) 0x7f2304134f90 con 0x7f230414be90 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:20:02.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.339+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23041433b0 con 0x7f230414be90 2024-03-25T18:20:02.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.340+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841166023 0 0) 0x7f23041395e0 con 0x7f23040b1730 2024-03-25T18:20:02.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.340+0000 7f22fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.340+0000 7f22fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.340+0000 7f22fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.340+0000 7f22fa7fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:20:02.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.340+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304134f90 con 0x7f23040b1730 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.344+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3170017645 0 0) 0x7f22e8004eb0 con 0x7f23040b4d60 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.344+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23041395e0 con 0x7f23040b4d60 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.344+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22f40019b0 con 0x7f23040b4d60 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.344+0000 7f22fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.344+0000 7f22fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.344+0000 7f22fa7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:02.345 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.344+0000 7f22fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:02.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.345+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834987314 0 0) 0x7f2304134f90 con 0x7f23040b1730 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.345+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22e8004eb0 con 0x7f23040b1730 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.345+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2300001560 con 0x7f23040b1730 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.345+0000 7f22fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.345+0000 7f22fa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.345+0000 7f22fa7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:02.348 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.345+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4113480112 0 0) 0x7f23041433b0 con 0x7f230414be90 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.346+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2304134f90 con 0x7f230414be90 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.346+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22fc001a50 con 0x7f230414be90 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.346+0000 7f22fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.346+0000 7f22fa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.346+0000 7f22fa7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:02.349 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:02.350 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:02.350 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.352 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.347+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3617079338 0 0) 0x7f23041395e0 con 0x7f23040b4d60 2024-03-25T18:20:02.352 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 >> v1:172.21.15.120:6789/0 conn(0x7f230414be90 legacy=0x7f2304134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.352 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 >> v1:172.21.15.42:6789/0 conn(0x7f23040b1730 legacy=0x7f2304142ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.352 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.352 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 1 monclient: found mon.c 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304139850 con 0x7f23040b4d60 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f22fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 10 monclient: _renew_subs 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2304139b30 con 0x7f23040b4d60 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 10 monclient: _renew_subs 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.348+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23041425c0 con 0x7f23040b4d60 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.349+0000 7f230bc93640 1 librados: init done 2024-03-25T18:20:02.354 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.350+0000 7f230bc93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.354 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.350+0000 7f230bc93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.354 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.350+0000 7f230bc93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:02.354 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.350+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2304062060 con 0x7f23040b4d60 2024-03-25T18:20:02.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.426+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22f4001b10 con 0x7f23040b4d60 2024-03-25T18:20:02.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.426+0000 7f22fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.427+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22f4004c30 con 0x7f23040b4d60 2024-03-25T18:20:02.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.427+0000 7f22fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.427+0000 7f22fbfff640 4 set_mon_vals no callback set 2024-03-25T18:20:02.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.427+0000 7f22fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:02.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.427+0000 7f22fa7fc640 10 monclient: dump: 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.427+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f22f401ac00 con 0x7f23040b4d60 2024-03-25T18:20:02.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.428+0000 7f2309a2f640 10 monclient: get_auth_request con 0x7f22e806e0b0 auth_method 0 2024-03-25T18:20:02.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.428+0000 7f22fa7fc640 10 client.4214.objecter ms_handle_connect 0x7f22e806e0b0 2024-03-25T18:20:02.438 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:17.484 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:02.544 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:17.617 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.629+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f22f40548b0 con 0x7f23040b4d60 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.630+0000 7f22fa7fc640 10 client.4214.objecter ms_dispatch 0x7f2304000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.630+0000 7f22fa7fc640 3 client.4214.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.630+0000 7f22fa7fc640 3 client.4214.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.630+0000 7f22fa7fc640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.630+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22f40503c0 con 0x7f23040b4d60 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.630+0000 7f22fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.630+0000 7f22fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:02.778 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:17.851 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:02.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.859+0000 7f230bc93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.859+0000 7f230bc93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.859+0000 7f230bc93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:02.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.859+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23040627b0 con 0x7f23040b4d60 2024-03-25T18:20:02.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.886+0000 7f22fa7fc640 1 -- 172.21.15.42:0/2524279107 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f22f40522f0 con 0x7f23040b4d60 2024-03-25T18:20:02.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.886+0000 7f22fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.886+0000 7f22fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:02.888 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:02.888 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:02.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 10 librados: watch_flush enter 2024-03-25T18:20:02.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 10 librados: watch_flush exit 2024-03-25T18:20:02.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-25T18:20:02.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 10 client.4214.objecter successfully canceled tick 2024-03-25T18:20:02.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 >> v1:172.21.15.42:6800/3081750888 conn(0x7f22e806e0b0 legacy=0x7f22e8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 10 monclient: shutdown 2024-03-25T18:20:02.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.891 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 >> v1:172.21.15.42:6790/0 conn(0x7f23040b4d60 legacy=0x7f2304134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.891 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 shutdown_connections 2024-03-25T18:20:02.891 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 >> 172.21.15.42:0/2524279107 conn(0x7f23040617a0 msgr2=0x7f23040b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.891 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 shutdown_connections 2024-03-25T18:20:02.891 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 1 -- 172.21.15.42:0/2524279107 wait complete. 2024-03-25T18:20:02.891 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:02.889+0000 7f230bc93640 1 librados: shutdown 2024-03-25T18:20:02.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:06.725 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:21.791 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:06.729 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:21.795 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.009 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:22.065 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.013 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:22.069 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.203 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:22.272 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.207 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:22.276 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.730 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:22.782 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.735 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:22.787 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.794 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:22.840 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:07.797 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:22.844 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:08.435 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:23.508 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:08.439 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:23.512 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:08.486 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:23.541 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:08.490 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:23.545 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:08.678 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:23.751 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:08.684 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:23.757 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:09.706 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:09.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.797+0000 7ff4f7e93640 10 monclient: get_monmap_and_config 2024-03-25T18:20:09.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.797+0000 7ff4f7e93640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.797+0000 7ff4f7e93640 10 monclient: monmap: 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:09.798313+0000 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:09.798313+0000 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.797+0000 7ff4f7e93640 1 Processor -- start 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 1 -- start start 2024-03-25T18:20:09.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient: init 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient(hunting): picked mon.noname-b con 0x7ff4f00b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient(hunting): picked mon.noname-c con 0x7ff4f00b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient(hunting): picked mon.noname-a con 0x7ff4f014be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f00af940 con 0x7ff4f014be90 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f00b0a70 con 0x7ff4f00b1730 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f00b0d50 con 0x7ff4f00b4d60 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.798+0000 7ff4f7e93640 10 monclient(hunting): authenticate will time out at 7325.025394s 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.799+0000 7ff4f5c2f640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ff4f00b4d60 0x7ff4f013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:39168/0 (socket says 172.21.15.42:39168) 2024-03-25T18:20:09.801 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.799+0000 7ff4f5c2f640 1 -- 172.21.15.42:0/3521273065 learned_addr learned my addr 172.21.15.42:0/3521273065 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.802+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162017222 0 0) 0x7ff4f00af940 con 0x7ff4f014be90 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.802+0000 7ff4f4c2d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.802+0000 7ff4f4c2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.802+0000 7ff4f4c2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.802+0000 7ff4f4c2d640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.802+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc003520 con 0x7ff4f014be90 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.803+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424478545 0 0) 0x7ff4f00b0d50 con 0x7ff4f00b4d60 2024-03-25T18:20:09.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.803+0000 7ff4f4c2d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.803+0000 7ff4f4c2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.803+0000 7ff4f4c2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.803+0000 7ff4f4c2d640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:20:09.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.803+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f00af940 con 0x7ff4f00b4d60 2024-03-25T18:20:09.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4210785521 0 0) 0x7ff4dc003520 con 0x7ff4f014be90 2024-03-25T18:20:09.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 >> v1:172.21.15.42:6790/0 conn(0x7ff4f00b4d60 legacy=0x7ff4f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 >> v1:172.21.15.120:6789/0 conn(0x7ff4f00b1730 legacy=0x7ff4f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 1 monclient: found mon.noname-a 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f00b1030 con 0x7ff4f014be90 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff4e8001a10 con 0x7ff4f014be90 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f4c2d640 10 monclient: dump: 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:09.809 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f7e93640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:20:09.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.806+0000 7ff4f7e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4e8001b70 con 0x7ff4f014be90 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f4c2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f4c2d640 1 -- 172.21.15.42:0/3521273065 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff4e8001f80 con 0x7ff4f014be90 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f4c2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f4c2d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f4c2d640 10 monclient: dump: 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:09.812 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f7e93640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.808+0000 7ff4f7e93640 4 set_mon_vals no callback set 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.809+0000 7ff4f7e93640 10 monclient: shutdown 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.809+0000 7ff4f7e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.809+0000 7ff4f7e93640 1 -- 172.21.15.42:0/3521273065 >> v1:172.21.15.42:6789/0 conn(0x7ff4f014be90 legacy=0x7ff4f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.809+0000 7ff4f7e93640 1 -- 172.21.15.42:0/3521273065 shutdown_connections 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.809+0000 7ff4f7e93640 1 -- 172.21.15.42:0/3521273065 >> 172.21.15.42:0/3521273065 conn(0x7ff4f00617a0 msgr2=0x7ff4f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.809+0000 7ff4f7e93640 1 -- 172.21.15.42:0/3521273065 shutdown_connections 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.809+0000 7ff4f7e93640 1 -- 172.21.15.42:0/3521273065 wait complete. 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.810+0000 7ff4f7e93640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.810+0000 7ff4f7e93640 10 monclient: monmap: 2024-03-25T18:20:09.813 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.810+0000 7ff4f7e93640 1 librados: starting msgr at 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.810+0000 7ff4f7e93640 1 librados: starting objecter 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 1 Processor -- start 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 1 -- start start 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 1 librados: setting wanted keys 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 1 librados: calling monclient init 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient: init 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient(hunting): picked mon.noname-a con 0x7ff4f00b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient(hunting): picked mon.noname-c con 0x7ff4f00b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient(hunting): picked mon.noname-b con 0x7ff4f014be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f0141f60 con 0x7ff4f00b1730 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f01421d0 con 0x7ff4f014be90 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f013bc30 con 0x7ff4f00b4d60 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.811+0000 7ff4f7e93640 10 monclient(hunting): authenticate will time out at 7325.038474s 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.812+0000 7ff4f542e640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ff4f00b4d60 0x7ff4f01d8230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:39176/0 (socket says 172.21.15.42:39176) 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.812+0000 7ff4f542e640 1 -- 172.21.15.42:0/2753901353 learned_addr learned my addr 172.21.15.42:0/2753901353 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.813+0000 7ff4e67fc640 10 client.?.objecter ms_handle_connect 0x7ff4f00b4d60 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.813+0000 7ff4e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.813+0000 7ff4e67fc640 10 client.?.objecter ms_handle_connect 0x7ff4f00b1730 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.813+0000 7ff4e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.815+0000 7ff4e67fc640 10 client.?.objecter ms_handle_connect 0x7ff4f014be90 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.815+0000 7ff4e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256752258 0 0) 0x7ff4f013bc30 con 0x7ff4f00b4d60 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d4004eb0 con 0x7ff4f00b4d60 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216880723 0 0) 0x7ff4f0141f60 con 0x7ff4f00b1730 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:20:09.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.816+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f013bc30 con 0x7ff4f00b1730 2024-03-25T18:20:09.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.818+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074043602 0 0) 0x7ff4f01421d0 con 0x7ff4f014be90 2024-03-25T18:20:09.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.818+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.818+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.818+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.818+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:20:09.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.818+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d4005360 con 0x7ff4f014be90 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.821+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1552214945 0 0) 0x7ff4f013bc30 con 0x7ff4f00b1730 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.821+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4f01421d0 con 0x7ff4f00b1730 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.821+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff4ec0018b0 con 0x7ff4f00b1730 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.821+0000 7ff4e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.821+0000 7ff4e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.821+0000 7ff4e67fc640 10 monclient(hunting): dump: 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.822 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.823 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:09.823 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:09.823 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:09.823 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:09.823 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:09.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.821+0000 7ff4e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:09.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.824+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964993671 0 0) 0x7ff4d4004eb0 con 0x7ff4f00b4d60 2024-03-25T18:20:09.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.824+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4f013bc30 con 0x7ff4f00b4d60 2024-03-25T18:20:09.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.824+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff4e0001960 con 0x7ff4f00b4d60 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.824+0000 7ff4e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.824+0000 7ff4e67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.824+0000 7ff4e67fc640 10 monclient(hunting): dump: 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:09.825 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2167006888 0 0) 0x7ff4f01421d0 con 0x7ff4f00b1730 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 >> v1:172.21.15.42:6790/0 conn(0x7ff4f00b4d60 legacy=0x7ff4f01d8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 >> v1:172.21.15.120:6789/0 conn(0x7ff4f014be90 legacy=0x7ff4f01e2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 1 monclient: found mon.a 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f01d54b0 con 0x7ff4f00b1730 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 5 monclient: authenticate success, global_id 4224 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 10 monclient: _renew_subs 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f0140d20 con 0x7ff4f00b1730 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 10 monclient: _renew_subs 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f01405f0 con 0x7ff4f00b1730 2024-03-25T18:20:09.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.825+0000 7ff4f7e93640 1 librados: init done 2024-03-25T18:20:09.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.826+0000 7ff4f7e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.826+0000 7ff4f7e93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.826+0000 7ff4f7e93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:09.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.826+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f00627b0 con 0x7ff4f00b1730 2024-03-25T18:20:09.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.842+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4ec0047e0 con 0x7ff4f00b1730 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.842+0000 7ff4e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.842+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff4ec001d80 con 0x7ff4f00b1730 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.842+0000 7ff4e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.842+0000 7ff4e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.842+0000 7ff4e67fc640 10 monclient: dump: 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:09.843 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:09.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.842+0000 7ff4e7fff640 4 set_mon_vals no callback set 2024-03-25T18:20:09.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.843+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff4ec01a9e0 con 0x7ff4f00b1730 2024-03-25T18:20:09.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.843+0000 7ff4f542e640 10 monclient: get_auth_request con 0x7ff4d406df10 auth_method 0 2024-03-25T18:20:09.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.844+0000 7ff4e67fc640 10 client.4224.objecter ms_handle_connect 0x7ff4d406df10 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.848+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7ff4ec054640 con 0x7ff4f00b1730 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.848+0000 7ff4e67fc640 10 client.4224.objecter ms_dispatch 0x7ff4f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.848+0000 7ff4e67fc640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.848+0000 7ff4e67fc640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:09.848+0000 7ff4e67fc640 20 client.4224.objecter dump_active .. 0 homeless 2024-03-25T18:20:10.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.169+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4ec04ff80 con 0x7ff4f00b1730 2024-03-25T18:20:10.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.169+0000 7ff4e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.169+0000 7ff4e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:10.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.352+0000 7ff4f7e93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.352+0000 7ff4f7e93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.352+0000 7ff4f7e93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:10.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.352+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4f00627b0 con 0x7ff4f00b1730 2024-03-25T18:20:10.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.375+0000 7ff4e67fc640 1 -- 172.21.15.42:0/2753901353 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7ff4ec04ff80 con 0x7ff4f00b1730 2024-03-25T18:20:10.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.376+0000 7ff4e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.376+0000 7ff4e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:10.377 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:10.377 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:10.379 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 10 librados: watch_flush enter 2024-03-25T18:20:10.379 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 10 librados: watch_flush exit 2024-03-25T18:20:10.379 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 20 client.4224.objecter shutdown clearing up homeless session... 2024-03-25T18:20:10.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 10 client.4224.objecter successfully canceled tick 2024-03-25T18:20:10.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff4d406df10 legacy=0x7ff4d40703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 10 monclient: shutdown 2024-03-25T18:20:10.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 >> v1:172.21.15.42:6789/0 conn(0x7ff4f00b1730 legacy=0x7ff4f01d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 shutdown_connections 2024-03-25T18:20:10.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.378+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 >> 172.21.15.42:0/2753901353 conn(0x7ff4f00617a0 msgr2=0x7ff4f013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.379+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 shutdown_connections 2024-03-25T18:20:10.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.379+0000 7ff4f7e93640 1 -- 172.21.15.42:0/2753901353 wait complete. 2024-03-25T18:20:10.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:10.379+0000 7ff4f7e93640 1 librados: shutdown 2024-03-25T18:20:10.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:10.873 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:00:25.939 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:10.930 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:00:25.986 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:11.283 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:00:26.352 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:11.883 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:00:26.935 35413== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:12.000 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:00:27.046 35422== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:12.709 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:00:27.764 35412== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:12.827 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:00:27.900 35410== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:13.280 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:00:28.353 35411== Warning: unimplemented fcntl command: 1038 2024-03-25T18:20:17.201 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:17.292 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.291+0000 7f8ee4e55640 10 monclient: get_monmap_and_config 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.291+0000 7f8ee4e55640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.291+0000 7f8ee4e55640 10 monclient: monmap: 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:17.292397+0000 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:17.292397+0000 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.293 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:17.294 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:17.294 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:17.294 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:17.294 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.294 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.291+0000 7f8ee4e55640 1 Processor -- start 2024-03-25T18:20:17.294 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 1 -- start start 2024-03-25T18:20:17.294 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient: init 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee00b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient(hunting): picked mon.noname-a con 0x7f8ee00b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient(hunting): picked mon.noname-b con 0x7f8ee014be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee00af940 con 0x7f8ee00b1730 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee00b0a70 con 0x7f8ee00b4d60 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee00b0d50 con 0x7f8ee014be90 2024-03-25T18:20:17.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.292+0000 7f8ee4e55640 10 monclient(hunting): authenticate will time out at 7332.519485s 2024-03-25T18:20:17.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.293+0000 7f8ede59c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8ee00b4d60 0x7f8ee013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:51220/0 (socket says 172.21.15.42:51220) 2024-03-25T18:20:17.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.293+0000 7f8ede59c640 1 -- 172.21.15.42:0/3253561161 learned_addr learned my addr 172.21.15.42:0/3253561161 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:17.301 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56904985 0 0) 0x7f8ee00b0a70 con 0x7f8ee00b4d60 2024-03-25T18:20:17.301 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.301 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.301 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): my global_id is 4228 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec8003520 con 0x7f8ee00b4d60 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241109406 0 0) 0x7f8ee00af940 con 0x7f8ee00b1730 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.296+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee00b0a70 con 0x7f8ee00b1730 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.297+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153965860 0 0) 0x7f8ee00b0d50 con 0x7f8ee014be90 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.297+0000 7f8edd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.297+0000 7f8edd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.297+0000 7f8edd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.297+0000 7f8edd59a640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.297+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee00af940 con 0x7f8ee014be90 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2578150054 0 0) 0x7f8ee00b0a70 con 0x7f8ee00b1730 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 >> v1:172.21.15.42:6790/0 conn(0x7f8ee014be90 legacy=0x7f8ee014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 >> v1:172.21.15.120:6789/0 conn(0x7f8ee00b4d60 legacy=0x7f8ee013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 1 monclient: found mon.noname-a 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.301+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee00b1030 con 0x7f8ee00b1730 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8edd59a640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8edd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8edd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8ed4001670 con 0x7f8ee00b1730 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8edd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8edd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8edd59a640 10 monclient: dump: 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:17.312 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8ee4e55640 5 monclient: authenticate success, global_id 4227 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.302+0000 7f8ee4e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ed40017d0 con 0x7f8ee00b1730 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8edd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8edd59a640 1 -- 172.21.15.42:0/3253561161 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8ed4001c20 con 0x7f8ee00b1730 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8ee4e55640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8ee4e55640 4 set_mon_vals no callback set 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8edd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8edd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:17.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.304+0000 7f8edd59a640 10 monclient: dump: 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:17.314 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.305+0000 7f8ee4e55640 10 monclient: shutdown 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.305+0000 7f8ee4e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.305+0000 7f8ee4e55640 1 -- 172.21.15.42:0/3253561161 >> v1:172.21.15.42:6789/0 conn(0x7f8ee00b1730 legacy=0x7f8ee00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.305+0000 7f8ee4e55640 1 -- 172.21.15.42:0/3253561161 shutdown_connections 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.305+0000 7f8ee4e55640 1 -- 172.21.15.42:0/3253561161 >> 172.21.15.42:0/3253561161 conn(0x7f8ee00617a0 msgr2=0x7f8ee0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.305+0000 7f8ee4e55640 1 -- 172.21.15.42:0/3253561161 shutdown_connections 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.305+0000 7f8ee4e55640 1 -- 172.21.15.42:0/3253561161 wait complete. 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.306+0000 7f8ee4e55640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.306+0000 7f8ee4e55640 10 monclient: monmap: 2024-03-25T18:20:17.315 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.306+0000 7f8ee4e55640 1 librados: starting msgr at 2024-03-25T18:20:17.316 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.306+0000 7f8ee4e55640 1 librados: starting objecter 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.306+0000 7f8ee4e55640 1 Processor -- start 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 1 -- start start 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 1 librados: setting wanted keys 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 1 librados: calling monclient init 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 10 monclient: init 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 10 monclient(hunting): picked mon.noname-b con 0x7f8ee00b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 10 monclient(hunting): picked mon.noname-a con 0x7f8ee00b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee014be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:17.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee01422d0 con 0x7f8ee00b4d60 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.307+0000 7f8ee4e55640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee0134f90 con 0x7f8ee00b1730 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8ee4e55640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee0139920 con 0x7f8ee014be90 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8ee4e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8ee4e55640 10 monclient(hunting): authenticate will time out at 7332.534582s 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8ede59c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8ee00b1730 0x7f8ee0140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:51224/0 (socket says 172.21.15.42:51224) 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8ede59c640 1 -- 172.21.15.42:0/4078497108 learned_addr learned my addr 172.21.15.42:0/4078497108 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8eceffd640 10 client.?.objecter ms_handle_connect 0x7f8ee00b1730 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8eceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8eceffd640 10 client.?.objecter ms_handle_connect 0x7f8ee00b4d60 2024-03-25T18:20:17.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.308+0000 7f8eceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.309+0000 7f8eceffd640 10 client.?.objecter ms_handle_connect 0x7f8ee014be90 2024-03-25T18:20:17.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.309+0000 7f8eceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.310+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887650982 0 0) 0x7f8ee0134f90 con 0x7f8ee00b1730 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.310+0000 7f8eceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.310+0000 7f8eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.310+0000 7f8eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.310+0000 7f8eceffd640 10 monclient(hunting): my global_id is 4231 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.310+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec0004eb0 con 0x7f8ee00b1730 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219387005 0 0) 0x7f8ee01422d0 con 0x7f8ee00b4d60 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): my global_id is 4230 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0134f90 con 0x7f8ee00b4d60 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533396290 0 0) 0x7f8ee0139920 con 0x7f8ee014be90 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.311+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee01422d0 con 0x7f8ee014be90 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.318+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3741839581 0 0) 0x7f8ec0004eb0 con 0x7f8ee00b1730 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.318+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ee0139920 con 0x7f8ee00b1730 2024-03-25T18:20:17.325 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.318+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8ed00013b0 con 0x7f8ee00b1730 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.318+0000 7f8eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.318+0000 7f8eceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.318+0000 7f8eceffd640 10 monclient(hunting): dump: 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:17.326 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.318+0000 7f8eceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350142828 0 0) 0x7f8ee01422d0 con 0x7f8ee014be90 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ec0004eb0 con 0x7f8ee014be90 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8ed8001bd0 con 0x7f8ee014be90 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 10 monclient(hunting): dump: 2024-03-25T18:20:17.327 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2266776444 0 0) 0x7f8ee0134f90 con 0x7f8ee00b4d60 2024-03-25T18:20:17.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ee01422d0 con 0x7f8ee00b4d60 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8ed4001ba0 con 0x7f8ee00b4d60 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.319+0000 7f8eceffd640 10 monclient(hunting): dump: 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.329 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:17.330 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:17.330 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:17.330 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:17.330 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.323+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1403840039 0 0) 0x7f8ec0004eb0 con 0x7f8ee014be90 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.323+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 >> v1:172.21.15.120:6789/0 conn(0x7f8ee00b1730 legacy=0x7f8ee0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.323+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 >> v1:172.21.15.42:6789/0 conn(0x7f8ee00b4d60 legacy=0x7f8ee0134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.323+0000 7f8eceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.323+0000 7f8eceffd640 1 monclient: found mon.c 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.323+0000 7f8eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8eceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee0139c00 con 0x7f8ee014be90 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8eceffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8eceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 5 monclient: authenticate success, global_id 4226 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 10 monclient: _renew_subs 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee0139ee0 con 0x7f8ee014be90 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 10 monclient: _renew_subs 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee0143310 con 0x7f8ee014be90 2024-03-25T18:20:17.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.324+0000 7f8ee4e55640 1 librados: init done 2024-03-25T18:20:17.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.325+0000 7f8ee4e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.325+0000 7f8ee4e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.325+0000 7f8ee4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:17.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.325+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee00afbd0 con 0x7f8ee014be90 2024-03-25T18:20:17.596 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.595+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ed8001fe0 con 0x7f8ee014be90 2024-03-25T18:20:17.596 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.595+0000 7f8eceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:17.596 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.595+0000 7f8edcd99640 4 set_mon_vals no callback set 2024-03-25T18:20:17.612 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.611+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8ed8003b30 con 0x7f8ee014be90 2024-03-25T18:20:17.612 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.611+0000 7f8eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.612 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.611+0000 7f8eceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:17.612 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.611+0000 7f8eceffd640 10 monclient: dump: 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:17.613 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:17.614 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.612+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f8ed8019b00 con 0x7f8ee014be90 2024-03-25T18:20:17.614 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.613+0000 7f8ede59c640 10 monclient: get_auth_request con 0x7f8ec006e0f0 auth_method 0 2024-03-25T18:20:17.614 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.613+0000 7f8eceffd640 10 client.4226.objecter ms_handle_connect 0x7f8ec006e0f0 2024-03-25T18:20:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f8ed80537b0 con 0x7f8ee014be90 2024-03-25T18:20:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 10 client.4226.objecter ms_dispatch 0x7f8ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 3 client.4226.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 3 client.4226.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 20 client.4226.objecter dump_active .. 0 homeless 2024-03-25T18:20:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ed804f0f0 con 0x7f8ee014be90 2024-03-25T18:20:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.620+0000 7f8eceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:17.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.813+0000 7f8ee4e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:17.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.813+0000 7f8ee4e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:17.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.813+0000 7f8ee4e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.813+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ee00627b0 con 0x7f8ee014be90 2024-03-25T18:20:17.919 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.918+0000 7f8eceffd640 1 -- 172.21.15.42:0/4078497108 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f8ed80511f0 con 0x7f8ee014be90 2024-03-25T18:20:17.919 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.918+0000 7f8eceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:17.919 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.918+0000 7f8eceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:17.920 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:17.920 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:17.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.920+0000 7f8ee4e55640 10 librados: watch_flush enter 2024-03-25T18:20:17.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.920+0000 7f8ee4e55640 10 librados: watch_flush exit 2024-03-25T18:20:17.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.920+0000 7f8ee4e55640 20 client.4226.objecter shutdown clearing up homeless session... 2024-03-25T18:20:17.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.920+0000 7f8ee4e55640 10 client.4226.objecter successfully canceled tick 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 >> v1:172.21.15.42:6800/3081750888 conn(0x7f8ec006e0f0 legacy=0x7f8ec00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 10 monclient: shutdown 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 >> v1:172.21.15.42:6790/0 conn(0x7f8ee014be90 legacy=0x7f8ee0134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 shutdown_connections 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 >> 172.21.15.42:0/4078497108 conn(0x7f8ee00617a0 msgr2=0x7f8ee00b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 shutdown_connections 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.921+0000 7f8ee4e55640 1 -- 172.21.15.42:0/4078497108 wait complete. 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:17.922+0000 7f8ee4e55640 1 librados: shutdown 2024-03-25T18:20:17.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:19.307 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-25T18:20:19.304+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:20:19.512 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-25T18:20:19.508+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:20:19.558 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-25T18:20:19.554+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:20:20.161 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-25T18:20:20.158+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:20:20.440 INFO:tasks.ceph.osd.3.smithi042.stderr:2024-03-25T18:20:20.436+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:20:21.842 INFO:tasks.ceph.osd.1.smithi042.stderr:2024-03-25T18:20:21.838+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:20:22.222 INFO:tasks.ceph.osd.0.smithi042.stderr:2024-03-25T18:20:22.220+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:20:23.101 INFO:tasks.ceph.osd.2.smithi042.stderr:2024-03-25T18:20:23.098+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:20:24.740 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:24.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.856+0000 7f0d30ab3640 10 monclient: get_monmap_and_config 2024-03-25T18:20:24.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.856+0000 7f0d30ab3640 10 monclient: build_initial_monmap 2024-03-25T18:20:24.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.856+0000 7f0d30ab3640 10 monclient: monmap: 2024-03-25T18:20:24.857 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:24.857789+0000 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:24.857789+0000 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:24.858 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:24.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.857+0000 7f0d30ab3640 1 Processor -- start 2024-03-25T18:20:24.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.857+0000 7f0d30ab3640 1 -- start start 2024-03-25T18:20:24.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.857+0000 7f0d30ab3640 10 monclient: init 2024-03-25T18:20:24.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:24.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 10 monclient(hunting): picked mon.noname-b con 0x7f0d2c0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 10 monclient(hunting): picked mon.noname-a con 0x7f0d2c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 10 monclient(hunting): picked mon.noname-c con 0x7f0d2c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c142520 con 0x7f0d2c13f710 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c0b28a0 con 0x7f0d2c1430c0 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c0b2a30 con 0x7f0d2c0a98f0 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:24.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.858+0000 7f0d30ab3640 10 monclient(hunting): authenticate will time out at 7340.084741s 2024-03-25T18:20:24.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.870+0000 7f0d2ad9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f0d2c1430c0 0x7f0d2c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:55142/0 (socket says 172.21.15.42:55142) 2024-03-25T18:20:24.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.870+0000 7f0d2ad9d640 1 -- 172.21.15.42:0/149585653 learned_addr learned my addr 172.21.15.42:0/149585653 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:24.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.881+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780462938 0 0) 0x7f0d2c0b28a0 con 0x7f0d2c1430c0 2024-03-25T18:20:24.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.881+0000 7f0d2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.881+0000 7f0d2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.881+0000 7f0d2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.881+0000 7f0d2959a640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:20:24.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.881+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14003520 con 0x7f0d2c1430c0 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712028068 0 0) 0x7f0d2c142520 con 0x7f0d2c13f710 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2c0b28a0 con 0x7f0d2c13f710 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337541996 0 0) 0x7f0d2c0b2a30 con 0x7f0d2c0a98f0 2024-03-25T18:20:24.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:20:24.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.886+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2c142520 con 0x7f0d2c0a98f0 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1473529190 0 0) 0x7f0d2c0b28a0 con 0x7f0d2c13f710 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 >> v1:172.21.15.42:6790/0 conn(0x7f0d2c0a98f0 legacy=0x7f0d2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 >> v1:172.21.15.120:6789/0 conn(0x7f0d2c1430c0 legacy=0x7f0d2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 1 monclient: found mon.noname-a 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c0b44c0 con 0x7f0d2c13f710 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 10 monclient: _finish_auth 0 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:24.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d20001670 con 0x7f0d2c13f710 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d2959a640 10 monclient: dump: 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d30ab3640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:20:24.901 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.898+0000 7f0d30ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d200017d0 con 0x7f0d2c13f710 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d2959a640 1 -- 172.21.15.42:0/149585653 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d20001be0 con 0x7f0d2c13f710 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d2959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d2959a640 10 monclient: dump: 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:24.917 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d30ab3640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:24.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.916+0000 7f0d30ab3640 4 set_mon_vals no callback set 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.917+0000 7f0d30ab3640 10 monclient: shutdown 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.917+0000 7f0d30ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.917+0000 7f0d30ab3640 1 -- 172.21.15.42:0/149585653 >> v1:172.21.15.42:6789/0 conn(0x7f0d2c13f710 legacy=0x7f0d2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.917+0000 7f0d30ab3640 1 -- 172.21.15.42:0/149585653 shutdown_connections 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.917+0000 7f0d30ab3640 1 -- 172.21.15.42:0/149585653 >> 172.21.15.42:0/149585653 conn(0x7f0d2c0b7ea0 msgr2=0x7f0d2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.917+0000 7f0d30ab3640 1 -- 172.21.15.42:0/149585653 shutdown_connections 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.917+0000 7f0d30ab3640 1 -- 172.21.15.42:0/149585653 wait complete. 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 10 monclient: build_initial_monmap 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 10 monclient: monmap: 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:24.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 1 librados: starting msgr at 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 1 librados: starting objecter 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 1 Processor -- start 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 1 -- start start 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 1 librados: setting wanted keys 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 1 librados: calling monclient init 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.918+0000 7f0d30ab3640 10 monclient: init 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 10 monclient(hunting): picked mon.noname-c con 0x7f0d2c0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 10 monclient(hunting): picked mon.noname-a con 0x7f0d2c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 10 monclient(hunting): picked mon.noname-b con 0x7f0d2c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c139190 con 0x7f0d2c13f710 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c137590 con 0x7f0d2c1430c0 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c138b40 con 0x7f0d2c0a98f0 2024-03-25T18:20:24.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:24.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.919+0000 7f0d30ab3640 10 monclient(hunting): authenticate will time out at 7340.145924s 2024-03-25T18:20:24.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.925+0000 7f0d2a59c640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f0d2c0a98f0 0x7f0d2c136e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:44918/0 (socket says 172.21.15.42:44918) 2024-03-25T18:20:24.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.925+0000 7f0d2a59c640 1 -- 172.21.15.42:0/4288927523 learned_addr learned my addr 172.21.15.42:0/4288927523 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:24.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.926+0000 7f0d1affd640 10 client.?.objecter ms_handle_connect 0x7f0d2c0a98f0 2024-03-25T18:20:24.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.926+0000 7f0d1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.930+0000 7f0d1affd640 10 client.?.objecter ms_handle_connect 0x7f0d2c1430c0 2024-03-25T18:20:24.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.930+0000 7f0d1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.933+0000 7f0d1affd640 10 client.?.objecter ms_handle_connect 0x7f0d2c13f710 2024-03-25T18:20:24.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.933+0000 7f0d1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.935+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302729689 0 0) 0x7f0d2c137590 con 0x7f0d2c1430c0 2024-03-25T18:20:24.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.935+0000 7f0d1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.935+0000 7f0d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.935+0000 7f0d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.935+0000 7f0d1affd640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:20:24.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.935+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c004eb0 con 0x7f0d2c1430c0 2024-03-25T18:20:24.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187693940 0 0) 0x7f0d2c139190 con 0x7f0d2c13f710 2024-03-25T18:20:24.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:20:24.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2c137590 con 0x7f0d2c13f710 2024-03-25T18:20:24.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679725403 0 0) 0x7f0d2c138b40 con 0x7f0d2c0a98f0 2024-03-25T18:20:24.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:20:24.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.943+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2c139190 con 0x7f0d2c0a98f0 2024-03-25T18:20:24.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.952+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4050162052 0 0) 0x7f0d0c004eb0 con 0x7f0d2c1430c0 2024-03-25T18:20:24.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.952+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d2c138b40 con 0x7f0d2c1430c0 2024-03-25T18:20:24.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.952+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d24001af0 con 0x7f0d2c1430c0 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.952+0000 7f0d1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.952+0000 7f0d1affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.952+0000 7f0d1affd640 10 monclient(hunting): dump: 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:24.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.952+0000 7f0d1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1535202317 0 0) 0x7f0d2c138b40 con 0x7f0d2c1430c0 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 >> v1:172.21.15.42:6790/0 conn(0x7f0d2c0a98f0 legacy=0x7f0d2c136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 >> v1:172.21.15.42:6789/0 conn(0x7f0d2c13f710 legacy=0x7f0d2c1364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 1 monclient: found mon.b 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c136bf0 con 0x7f0d2c1430c0 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 10 monclient: _finish_auth 0 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 10 monclient: _renew_subs 2024-03-25T18:20:24.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2c13afe0 con 0x7f0d2c1430c0 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 10 monclient: _renew_subs 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2c139620 con 0x7f0d2c1430c0 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.961+0000 7f0d30ab3640 1 librados: init done 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.962+0000 7f0d30ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.962+0000 7f0d30ab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.962+0000 7f0d30ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:24.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:24.962+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2c138f60 con 0x7f0d2c1430c0 2024-03-25T18:20:25.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.251+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d24001f20 con 0x7f0d2c1430c0 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.251+0000 7f0d1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.251+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d24004b60 con 0x7f0d2c1430c0 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.251+0000 7f0d1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.251+0000 7f0d1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.251+0000 7f0d1affd640 10 monclient: dump: 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:25.253 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:25.254 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:25.254 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:25.254 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.251+0000 7f0d28d99640 4 set_mon_vals no callback set 2024-03-25T18:20:25.254 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.252+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f0d24004d40 con 0x7f0d2c1430c0 2024-03-25T18:20:25.254 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.253+0000 7f0d2a59c640 10 monclient: get_auth_request con 0x7f0d0c06ddf0 auth_method 0 2024-03-25T18:20:25.254 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.253+0000 7f0d1affd640 10 client.4261.objecter ms_handle_connect 0x7f0d0c06ddf0 2024-03-25T18:20:25.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.258+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f0d24054830 con 0x7f0d2c1430c0 2024-03-25T18:20:25.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.258+0000 7f0d1affd640 10 client.4261.objecter ms_dispatch 0x7f0d2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:25.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.258+0000 7f0d1affd640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:25.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.258+0000 7f0d1affd640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:25.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.258+0000 7f0d1affd640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:20:25.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.259+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d24050340 con 0x7f0d2c1430c0 2024-03-25T18:20:25.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.259+0000 7f0d1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.259+0000 7f0d1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.479+0000 7f0d30ab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.479+0000 7f0d30ab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.479+0000 7f0d30ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.479+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d2c0627b0 con 0x7f0d2c1430c0 2024-03-25T18:20:25.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.532+0000 7f0d1affd640 1 -- 172.21.15.42:0/4288927523 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f0d24052270 con 0x7f0d2c1430c0 2024-03-25T18:20:25.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.533+0000 7f0d1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.533+0000 7f0d1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:25.534 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:25.534 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:25.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 10 librados: watch_flush enter 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 10 librados: watch_flush exit 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 >> v1:172.21.15.42:6800/3081750888 conn(0x7f0d0c06ddf0 legacy=0x7f0d0c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 10 monclient: shutdown 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 >> v1:172.21.15.120:6789/0 conn(0x7f0d2c1430c0 legacy=0x7f0d2c1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 shutdown_connections 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 >> 172.21.15.42:0/4288927523 conn(0x7f0d2c0b7ea0 msgr2=0x7f0d2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 shutdown_connections 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 1 -- 172.21.15.42:0/4288927523 wait complete. 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:25.535+0000 7f0d30ab3640 1 librados: shutdown 2024-03-25T18:20:25.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:32.355 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:32.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.473+0000 7fbe27382640 10 monclient: get_monmap_and_config 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.473+0000 7fbe27382640 10 monclient: build_initial_monmap 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.473+0000 7fbe27382640 10 monclient: monmap: 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:32.474474+0000 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:32.474474+0000 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:32.475 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:32.476 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:32.476 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:32.476 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:32.476 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.474+0000 7fbe27382640 1 Processor -- start 2024-03-25T18:20:32.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.474+0000 7fbe27382640 1 -- start start 2024-03-25T18:20:32.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.474+0000 7fbe27382640 10 monclient: init 2024-03-25T18:20:32.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:32.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:32.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 10 monclient(hunting): picked mon.noname-c con 0x7fbe200a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:32.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:32.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 10 monclient(hunting): picked mon.noname-a con 0x7fbe201430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:32.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe20142520 con 0x7fbe201430c0 2024-03-25T18:20:32.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe200b28a0 con 0x7fbe200a98f0 2024-03-25T18:20:32.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe200b2a30 con 0x7fbe2013f710 2024-03-25T18:20:32.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:32.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.475+0000 7fbe27382640 10 monclient(hunting): authenticate will time out at 7347.701899s 2024-03-25T18:20:32.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.481+0000 7fbe2511e640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fbe200a98f0 0x7fbe200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:36942/0 (socket says 172.21.15.42:36942) 2024-03-25T18:20:32.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.481+0000 7fbe2511e640 1 -- 172.21.15.42:0/1502017712 learned_addr learned my addr 172.21.15.42:0/1502017712 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:32.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.492+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551753175 0 0) 0x7fbe200b28a0 con 0x7fbe200a98f0 2024-03-25T18:20:32.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.492+0000 7fbe17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.492+0000 7fbe17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.492+0000 7fbe17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.492+0000 7fbe17fff640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:20:32.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.492+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe0c003520 con 0x7fbe200a98f0 2024-03-25T18:20:32.496 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.495+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002738565 0 0) 0x7fbe20142520 con 0x7fbe201430c0 2024-03-25T18:20:32.496 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.495+0000 7fbe17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.496 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.495+0000 7fbe17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.496 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.495+0000 7fbe17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.496 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.495+0000 7fbe17fff640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:20:32.497 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.495+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe200b28a0 con 0x7fbe201430c0 2024-03-25T18:20:32.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.517+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506556590 0 0) 0x7fbe200b2a30 con 0x7fbe2013f710 2024-03-25T18:20:32.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.517+0000 7fbe17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.517+0000 7fbe17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.517+0000 7fbe17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.517+0000 7fbe17fff640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:20:32.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.517+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe20142520 con 0x7fbe2013f710 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.518+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696934762 0 0) 0x7fbe200b28a0 con 0x7fbe201430c0 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.518+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 >> v1:172.21.15.42:6790/0 conn(0x7fbe2013f710 legacy=0x7fbe201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.518+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 >> v1:172.21.15.120:6789/0 conn(0x7fbe200a98f0 legacy=0x7fbe200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.518+0000 7fbe17fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.518+0000 7fbe17fff640 1 monclient: found mon.noname-a 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.518+0000 7fbe17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:20:32.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe200b44c0 con 0x7fbe201430c0 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe27382640 5 monclient: authenticate success, global_id 4263 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbe180019f0 con 0x7fbe201430c0 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe17fff640 10 monclient: dump: 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:32.521 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.519+0000 7fbe27382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.529+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe18001b50 con 0x7fbe201430c0 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.529+0000 7fbe17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.529+0000 7fbe17fff640 1 -- 172.21.15.42:0/1502017712 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbe18001f60 con 0x7fbe201430c0 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.529+0000 7fbe17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.529+0000 7fbe17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.529+0000 7fbe17fff640 10 monclient: dump: 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.531 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.530+0000 7fbe27382640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:32.532 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.530+0000 7fbe27382640 4 set_mon_vals no callback set 2024-03-25T18:20:32.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.530+0000 7fbe27382640 10 monclient: shutdown 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.530+0000 7fbe27382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.530+0000 7fbe27382640 1 -- 172.21.15.42:0/1502017712 >> v1:172.21.15.42:6789/0 conn(0x7fbe201430c0 legacy=0x7fbe2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.531+0000 7fbe27382640 1 -- 172.21.15.42:0/1502017712 shutdown_connections 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.531+0000 7fbe27382640 1 -- 172.21.15.42:0/1502017712 >> 172.21.15.42:0/1502017712 conn(0x7fbe200b7ea0 msgr2=0x7fbe200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.531+0000 7fbe27382640 1 -- 172.21.15.42:0/1502017712 shutdown_connections 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.531+0000 7fbe27382640 1 -- 172.21.15.42:0/1502017712 wait complete. 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 10 monclient: build_initial_monmap 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 10 monclient: monmap: 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:32.535 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 1 librados: starting msgr at 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 1 librados: starting objecter 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 1 Processor -- start 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 1 -- start start 2024-03-25T18:20:32.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 1 librados: setting wanted keys 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 1 librados: calling monclient init 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.532+0000 7fbe27382640 10 monclient: init 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 10 monclient(hunting): picked mon.noname-a con 0x7fbe200a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 10 monclient(hunting): picked mon.noname-b con 0x7fbe201430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe201d48f0 con 0x7fbe200a98f0 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe201397d0 con 0x7fbe201430c0 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe201d4b50 con 0x7fbe2013f710 2024-03-25T18:20:32.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:32.538 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.533+0000 7fbe27382640 10 monclient(hunting): authenticate will time out at 7347.759986s 2024-03-25T18:20:32.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.539+0000 7fbe2491d640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fbe2013f710 0x7fbe200aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:36258/0 (socket says 172.21.15.42:36258) 2024-03-25T18:20:32.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.539+0000 7fbe2491d640 1 -- 172.21.15.42:0/3569876758 learned_addr learned my addr 172.21.15.42:0/3569876758 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:32.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.541+0000 7fbe15ffb640 10 client.?.objecter ms_handle_connect 0x7fbe200a98f0 2024-03-25T18:20:32.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.541+0000 7fbe15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.543+0000 7fbe15ffb640 10 client.?.objecter ms_handle_connect 0x7fbe2013f710 2024-03-25T18:20:32.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.543+0000 7fbe15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.551 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.550+0000 7fbe15ffb640 10 client.?.objecter ms_handle_connect 0x7fbe201430c0 2024-03-25T18:20:32.551 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.550+0000 7fbe15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.552+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697412077 0 0) 0x7fbe201d48f0 con 0x7fbe200a98f0 2024-03-25T18:20:32.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.552+0000 7fbe15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.552+0000 7fbe15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.552+0000 7fbe15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.552+0000 7fbe15ffb640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:20:32.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.552+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe04004eb0 con 0x7fbe200a98f0 2024-03-25T18:20:32.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.553+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658673592 0 0) 0x7fbe201d4b50 con 0x7fbe2013f710 2024-03-25T18:20:32.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.553+0000 7fbe15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.553+0000 7fbe15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.553+0000 7fbe15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.553+0000 7fbe15ffb640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:20:32.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.553+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe201d48f0 con 0x7fbe2013f710 2024-03-25T18:20:32.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.559+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146518086 0 0) 0x7fbe201397d0 con 0x7fbe201430c0 2024-03-25T18:20:32.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.559+0000 7fbe15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.559+0000 7fbe15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.559+0000 7fbe15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.559+0000 7fbe15ffb640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:20:32.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.559+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe201d4b50 con 0x7fbe201430c0 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.568+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3094011325 0 0) 0x7fbe04004eb0 con 0x7fbe200a98f0 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.568+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe201397d0 con 0x7fbe200a98f0 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.568+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbe1c001950 con 0x7fbe200a98f0 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.569+0000 7fbe15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.569+0000 7fbe15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.569+0000 7fbe15ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:32.570 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.569+0000 7fbe15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:32.588 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.587+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857006466 0 0) 0x7fbe201d4b50 con 0x7fbe201430c0 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.587+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe04004eb0 con 0x7fbe201430c0 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.587+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbe18001e60 con 0x7fbe201430c0 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.587+0000 7fbe15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.587+0000 7fbe15ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.587+0000 7fbe15ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:32.589 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:32.590 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:32.590 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:32.590 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.589+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251511024 0 0) 0x7fbe201d48f0 con 0x7fbe2013f710 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.589+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe201d4b50 con 0x7fbe2013f710 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.589+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbe10001a80 con 0x7fbe2013f710 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.589+0000 7fbe15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.589+0000 7fbe15ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.589+0000 7fbe15ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:32.591 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:32.592 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:32.592 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1073497648 0 0) 0x7fbe201397d0 con 0x7fbe200a98f0 2024-03-25T18:20:32.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 >> v1:172.21.15.42:6790/0 conn(0x7fbe2013f710 legacy=0x7fbe200aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 >> v1:172.21.15.120:6789/0 conn(0x7fbe201430c0 legacy=0x7fbe201d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:32.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 1 monclient: found mon.a 2024-03-25T18:20:32.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe200b16b0 con 0x7fbe200a98f0 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe27382640 5 monclient: authenticate success, global_id 4266 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe27382640 10 monclient: _renew_subs 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe27382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.598+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe200b0c10 con 0x7fbe200a98f0 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.599+0000 7fbe27382640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.599+0000 7fbe27382640 10 monclient: _renew_subs 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.599+0000 7fbe27382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.599+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe200b0610 con 0x7fbe200a98f0 2024-03-25T18:20:32.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.599+0000 7fbe27382640 1 librados: init done 2024-03-25T18:20:32.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.600+0000 7fbe27382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.600+0000 7fbe27382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.600+0000 7fbe27382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:32.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.600+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe201d4ce0 con 0x7fbe200a98f0 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.908+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe1c001d60 con 0x7fbe200a98f0 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.908+0000 7fbe15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.908+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbe1c004980 con 0x7fbe200a98f0 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.908+0000 7fbe15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.908+0000 7fbe15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.908+0000 7fbe15ffb640 10 monclient: dump: 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.909 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:32.910 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.910 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:32.910 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:32.910 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:32.910 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:32.910 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:32.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.908+0000 7fbe177fe640 4 set_mon_vals no callback set 2024-03-25T18:20:32.911 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.909+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fbe1c01a9d0 con 0x7fbe200a98f0 2024-03-25T18:20:32.911 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.909+0000 7fbe2491d640 10 monclient: get_auth_request con 0x7fbe0406e0b0 auth_method 0 2024-03-25T18:20:32.911 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.909+0000 7fbe15ffb640 10 client.4266.objecter ms_handle_connect 0x7fbe0406e0b0 2024-03-25T18:20:32.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fbe1c054780 con 0x7fbe200a98f0 2024-03-25T18:20:32.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 10 client.4266.objecter ms_dispatch 0x7fbe20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:32.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:32.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:32.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-25T18:20:32.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe1c0500c0 con 0x7fbe200a98f0 2024-03-25T18:20:32.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.916 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:32.914+0000 7fbe15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:33.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.147+0000 7fbe27382640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.147+0000 7fbe27382640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.147+0000 7fbe27382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:33.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.147+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe200627b0 con 0x7fbe200a98f0 2024-03-25T18:20:33.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.181+0000 7fbe15ffb640 1 -- 172.21.15.42:0/3569876758 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fbe1c0521c0 con 0x7fbe200a98f0 2024-03-25T18:20:33.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.181+0000 7fbe15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.181+0000 7fbe15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:33.182 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:33.183 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:33.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 10 librados: watch_flush enter 2024-03-25T18:20:33.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 10 librados: watch_flush exit 2024-03-25T18:20:33.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-25T18:20:33.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 10 client.4266.objecter successfully canceled tick 2024-03-25T18:20:33.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 >> v1:172.21.15.42:6800/3081750888 conn(0x7fbe0406e0b0 legacy=0x7fbe04070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 10 monclient: shutdown 2024-03-25T18:20:33.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:33.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 >> v1:172.21.15.42:6789/0 conn(0x7fbe200a98f0 legacy=0x7fbe201d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 shutdown_connections 2024-03-25T18:20:33.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 >> 172.21.15.42:0/3569876758 conn(0x7fbe200b7ea0 msgr2=0x7fbe20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:33.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 shutdown_connections 2024-03-25T18:20:33.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 1 -- 172.21.15.42:0/3569876758 wait complete. 2024-03-25T18:20:33.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:33.184+0000 7fbe27382640 1 librados: shutdown 2024-03-25T18:20:33.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:36.325 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:19.495 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.394 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:19.563 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.431 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:19.552 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.494 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:19.614 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.532 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:19.702 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.629 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:19.798 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.641 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:19.811 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.706 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:19.827 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:36.904 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:20.074 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:40.003 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:40.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.118+0000 7f481eefb640 10 monclient: get_monmap_and_config 2024-03-25T18:20:40.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.118+0000 7f481eefb640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.118+0000 7f481eefb640 10 monclient: monmap: 2024-03-25T18:20:40.119 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:40.119 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:40.119 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:40.119579+0000 2024-03-25T18:20:40.119 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:40.119579+0000 2024-03-25T18:20:40.120 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.120 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:40.120 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:40.120 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:40.120 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:40.120 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.119+0000 7f481eefb640 1 Processor -- start 2024-03-25T18:20:40.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.119+0000 7f481eefb640 1 -- start start 2024-03-25T18:20:40.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.119+0000 7f481eefb640 10 monclient: init 2024-03-25T18:20:40.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:40.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 10 monclient(hunting): picked mon.noname-c con 0x7f48180a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:40.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 10 monclient(hunting): picked mon.noname-b con 0x7f481813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:40.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 10 monclient(hunting): picked mon.noname-a con 0x7f48181430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:40.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4818142520 con 0x7f48181430c0 2024-03-25T18:20:40.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180b28a0 con 0x7f48180a98f0 2024-03-25T18:20:40.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180b2a30 con 0x7f481813f710 2024-03-25T18:20:40.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.120+0000 7f481eefb640 10 monclient(hunting): authenticate will time out at 7355.346720s 2024-03-25T18:20:40.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.134+0000 7f481cc97640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f48180a98f0 0x7f48180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:43532/0 (socket says 172.21.15.42:43532) 2024-03-25T18:20:40.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.134+0000 7f481cc97640 1 -- 172.21.15.42:0/322708521 learned_addr learned my addr 172.21.15.42:0/322708521 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:40.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.140+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613924199 0 0) 0x7f48180b28a0 con 0x7f48180a98f0 2024-03-25T18:20:40.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.140+0000 7f48177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.140+0000 7f48177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.140+0000 7f48177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.140+0000 7f48177fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:20:40.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.140+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4808003520 con 0x7f48180a98f0 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3729710345 0 0) 0x7f4808003520 con 0x7f48180a98f0 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 >> v1:172.21.15.42:6790/0 conn(0x7f481813f710 legacy=0x7f48181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 >> v1:172.21.15.42:6789/0 conn(0x7f48181430c0 legacy=0x7f481814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 1 monclient: found mon.noname-c 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48180b44c0 con 0x7f48180a98f0 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f480c001690 con 0x7f48180a98f0 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f48177fe640 10 monclient: dump: 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f481eefb640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:20:40.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.147+0000 7f481eefb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f480c0017f0 con 0x7f48180a98f0 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f48177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f48177fe640 1 -- 172.21.15.42:0/322708521 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f480c001c40 con 0x7f48180a98f0 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f48177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f48177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f48177fe640 10 monclient: dump: 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:40.171 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:40.172 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:40.172 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f481eefb640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:40.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.170+0000 7f481eefb640 4 set_mon_vals no callback set 2024-03-25T18:20:40.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.171+0000 7f481eefb640 10 monclient: shutdown 2024-03-25T18:20:40.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.171+0000 7f481eefb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:40.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.171+0000 7f481eefb640 1 -- 172.21.15.42:0/322708521 >> v1:172.21.15.120:6789/0 conn(0x7f48180a98f0 legacy=0x7f48180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.171+0000 7f481eefb640 1 -- 172.21.15.42:0/322708521 shutdown_connections 2024-03-25T18:20:40.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.171+0000 7f481eefb640 1 -- 172.21.15.42:0/322708521 >> 172.21.15.42:0/322708521 conn(0x7f48180b7ea0 msgr2=0x7f48180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:40.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.171+0000 7f481eefb640 1 -- 172.21.15.42:0/322708521 shutdown_connections 2024-03-25T18:20:40.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.171+0000 7f481eefb640 1 -- 172.21.15.42:0/322708521 wait complete. 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 10 monclient: monmap: 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 1 librados: starting msgr at 2024-03-25T18:20:40.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 1 librados: starting objecter 2024-03-25T18:20:40.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 1 Processor -- start 2024-03-25T18:20:40.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 1 -- start start 2024-03-25T18:20:40.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 1 librados: setting wanted keys 2024-03-25T18:20:40.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 1 librados: calling monclient init 2024-03-25T18:20:40.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.172+0000 7f481eefb640 10 monclient: init 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 10 monclient(hunting): picked mon.noname-c con 0x7f48180a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 10 monclient(hunting): picked mon.noname-a con 0x7f481813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 10 monclient(hunting): picked mon.noname-b con 0x7f48181430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4818138b20 con 0x7f481813f710 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4818137a80 con 0x7f48181430c0 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4818139880 con 0x7f48180a98f0 2024-03-25T18:20:40.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.173+0000 7f481eefb640 10 monclient(hunting): authenticate will time out at 7355.400329s 2024-03-25T18:20:40.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.187+0000 7f4817fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f481813f710 0x7f4818139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48402/0 (socket says 172.21.15.42:48402) 2024-03-25T18:20:40.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.187+0000 7f4817fff640 1 -- 172.21.15.42:0/496129788 learned_addr learned my addr 172.21.15.42:0/496129788 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:40.191 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.190+0000 7f48157fa640 10 client.?.objecter ms_handle_connect 0x7f481813f710 2024-03-25T18:20:40.191 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.190+0000 7f48157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.191+0000 7f48157fa640 10 client.?.objecter ms_handle_connect 0x7f48181430c0 2024-03-25T18:20:40.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.191+0000 7f48157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.192+0000 7f48157fa640 10 client.?.objecter ms_handle_connect 0x7f48180a98f0 2024-03-25T18:20:40.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.192+0000 7f48157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176882524 0 0) 0x7f4818137a80 con 0x7f48181430c0 2024-03-25T18:20:40.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47fc004eb0 con 0x7f48181430c0 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468005916 0 0) 0x7f4818138b20 con 0x7f481813f710 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.208+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4818137a80 con 0x7f481813f710 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.209+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494438893 0 0) 0x7f4818139880 con 0x7f48180a98f0 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.209+0000 7f48157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.209+0000 7f48157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.209+0000 7f48157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.209+0000 7f48157fa640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:20:40.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.209+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4818138b20 con 0x7f48180a98f0 2024-03-25T18:20:40.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.221+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450221221 0 0) 0x7f4818137a80 con 0x7f481813f710 2024-03-25T18:20:40.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.221+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4818139880 con 0x7f481813f710 2024-03-25T18:20:40.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.221+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f48040019b0 con 0x7f481813f710 2024-03-25T18:20:40.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.221+0000 7f48157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.221+0000 7f48157fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:40.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.221+0000 7f48157fa640 10 monclient(hunting): dump: 2024-03-25T18:20:40.222 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.223 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.221+0000 7f48157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:40.226 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.225+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 695326315 0 0) 0x7f47fc004eb0 con 0x7f48181430c0 2024-03-25T18:20:40.226 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.225+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4818137a80 con 0x7f48181430c0 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.225+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4810001b20 con 0x7f48181430c0 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.225+0000 7f48157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.225+0000 7f48157fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.225+0000 7f48157fa640 10 monclient(hunting): dump: 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:40.227 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.238+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1526456365 0 0) 0x7f4818138b20 con 0x7f48180a98f0 2024-03-25T18:20:40.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.238+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47fc004eb0 con 0x7f48180a98f0 2024-03-25T18:20:40.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.238+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f480c001b00 con 0x7f48180a98f0 2024-03-25T18:20:40.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.238+0000 7f48157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.238+0000 7f48157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.238+0000 7f48157fa640 10 monclient(hunting): dump: 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:40.240 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549555332 0 0) 0x7f4818139880 con 0x7f481813f710 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 >> v1:172.21.15.42:6790/0 conn(0x7f48180a98f0 legacy=0x7f4818137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 >> v1:172.21.15.120:6789/0 conn(0x7f48181430c0 legacy=0x7f48181e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 1 monclient: found mon.a 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4818139a10 con 0x7f481813f710 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f48157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 5 monclient: authenticate success, global_id 4272 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 10 monclient: _renew_subs 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:40.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48181dab00 con 0x7f481813f710 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 10 monclient: _renew_subs 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f481813b4f0 con 0x7f481813f710 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.246+0000 7f481eefb640 1 librados: init done 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.247+0000 7f481eefb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.247+0000 7f481eefb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.247+0000 7f481eefb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:40.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.247+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4818074cc0 con 0x7f481813f710 2024-03-25T18:20:40.615 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.614+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4804003610 con 0x7f481813f710 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.614+0000 7f48157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.614+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f48040039a0 con 0x7f481813f710 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.614+0000 7f48157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.614+0000 7f48157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.614+0000 7f48157fa640 10 monclient: dump: 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.616 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:40.617 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:40.617 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:40.617 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:40.617 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:40.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.614+0000 7f4816ffd640 4 set_mon_vals no callback set 2024-03-25T18:20:40.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.615+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f4804003bc0 con 0x7f481813f710 2024-03-25T18:20:40.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.616+0000 7f481cc97640 10 monclient: get_auth_request con 0x7f47fc06e0f0 auth_method 0 2024-03-25T18:20:40.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.616+0000 7f48157fa640 10 client.4272.objecter ms_handle_connect 0x7f47fc06e0f0 2024-03-25T18:20:40.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f4804054810 con 0x7f481813f710 2024-03-25T18:20:40.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 10 client.4272.objecter ms_dispatch 0x7f4818000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:40.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:40.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:40.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T18:20:40.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4804050320 con 0x7f481813f710 2024-03-25T18:20:40.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.623+0000 7f48157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:40.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.838+0000 7f481eefb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.838+0000 7f481eefb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.838+0000 7f481eefb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:20:40.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.838+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48180b3e80 con 0x7f481813f710 2024-03-25T18:20:40.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.949+0000 7f48157fa640 1 -- 172.21.15.42:0/496129788 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f4804052250 con 0x7f481813f710 2024-03-25T18:20:40.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.949+0000 7f48157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.949+0000 7f48157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:40.950 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:40.950 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 10 librados: watch_flush enter 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 10 librados: watch_flush exit 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 10 client.4272.objecter successfully canceled tick 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 >> v1:172.21.15.42:6800/3081750888 conn(0x7f47fc06e0f0 legacy=0x7f47fc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 10 monclient: shutdown 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.951+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 >> v1:172.21.15.42:6789/0 conn(0x7f481813f710 legacy=0x7f4818139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.952+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 shutdown_connections 2024-03-25T18:20:40.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.952+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 >> 172.21.15.42:0/496129788 conn(0x7f48180b7ea0 msgr2=0x7f4818074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:40.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.952+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 shutdown_connections 2024-03-25T18:20:40.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.952+0000 7f481eefb640 1 -- 172.21.15.42:0/496129788 wait complete. 2024-03-25T18:20:40.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:40.952+0000 7f481eefb640 1 librados: shutdown 2024-03-25T18:20:40.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:47.770 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:47.885 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.884+0000 7f4b116c9640 10 monclient: get_monmap_and_config 2024-03-25T18:20:47.885 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.884+0000 7f4b116c9640 10 monclient: build_initial_monmap 2024-03-25T18:20:47.885 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.884+0000 7f4b116c9640 10 monclient: monmap: 2024-03-25T18:20:47.885 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:47.885 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:47.885 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:47.885503+0000 2024-03-25T18:20:47.886 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:47.885503+0000 2024-03-25T18:20:47.886 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:47.886 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:47.886 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:47.886 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:47.886 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:47.886 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.885+0000 7f4b116c9640 1 Processor -- start 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.885+0000 7f4b116c9640 1 -- start start 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.885+0000 7f4b116c9640 10 monclient: init 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4b0c0a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:47.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4b0c1430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:47.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c142520 con 0x7f4b0c13f710 2024-03-25T18:20:47.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0b28a0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0b2a30 con 0x7f4b0c1430c0 2024-03-25T18:20:47.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:47.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.886+0000 7f4b116c9640 10 monclient(hunting): authenticate will time out at 7363.112956s 2024-03-25T18:20:47.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.902+0000 7f4b0a7fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4b0c13f710 0x7f4b0c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44514/0 (socket says 172.21.15.42:44514) 2024-03-25T18:20:47.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.902+0000 7f4b0a7fc640 1 -- 172.21.15.42:0/875630530 learned_addr learned my addr 172.21.15.42:0/875630530 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:47.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936600442 0 0) 0x7f4b0c142520 con 0x7f4b0c13f710 2024-03-25T18:20:47.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:20:47.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af4003520 con 0x7f4b0c13f710 2024-03-25T18:20:47.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121763009 0 0) 0x7f4b0c0b28a0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.924+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c142520 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3102821378 0 0) 0x7f4b0c142520 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 >> v1:172.21.15.42:6790/0 conn(0x7f4b0c1430c0 legacy=0x7f4b0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 >> v1:172.21.15.42:6789/0 conn(0x7f4b0c13f710 legacy=0x7f4b0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 1 monclient: found mon.noname-c 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c0b44c0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4afc001670 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b09ffb640 10 monclient: dump: 2024-03-25T18:20:47.937 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b116c9640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:20:47.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.935+0000 7f4b116c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4afc0017d0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b09ffb640 1 -- 172.21.15.42:0/875630530 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4afc001ba0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b116c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b116c9640 4 set_mon_vals no callback set 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b09ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.947+0000 7f4b09ffb640 10 monclient: dump: 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.948+0000 7f4b116c9640 10 monclient: shutdown 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.948+0000 7f4b116c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.948+0000 7f4b116c9640 1 -- 172.21.15.42:0/875630530 >> v1:172.21.15.120:6789/0 conn(0x7f4b0c0a98f0 legacy=0x7f4b0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.948+0000 7f4b116c9640 1 -- 172.21.15.42:0/875630530 shutdown_connections 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.948+0000 7f4b116c9640 1 -- 172.21.15.42:0/875630530 >> 172.21.15.42:0/875630530 conn(0x7f4b0c0b7ea0 msgr2=0x7f4b0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.948+0000 7f4b116c9640 1 -- 172.21.15.42:0/875630530 shutdown_connections 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.948+0000 7f4b116c9640 1 -- 172.21.15.42:0/875630530 wait complete. 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.949+0000 7f4b116c9640 10 monclient: build_initial_monmap 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.949+0000 7f4b116c9640 10 monclient: monmap: 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:47.952 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.949+0000 7f4b116c9640 1 librados: starting msgr at 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.949+0000 7f4b116c9640 1 librados: starting objecter 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.949+0000 7f4b116c9640 1 Processor -- start 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 1 -- start start 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 1 librados: setting wanted keys 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 1 librados: calling monclient init 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient: init 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:47.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient(hunting): picked mon.noname-b con 0x7f4b0c0a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:47.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:47.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient(hunting): picked mon.noname-c con 0x7f4b0c1430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:47.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0b1880 con 0x7f4b0c13f710 2024-03-25T18:20:47.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0abe70 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0ac150 con 0x7f4b0c1430c0 2024-03-25T18:20:47.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:47.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.950+0000 7f4b116c9640 10 monclient(hunting): authenticate will time out at 7363.177451s 2024-03-25T18:20:47.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.963+0000 7f4b0b7fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f4b0c1430c0 0x7f4b0c1e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:37694/0 (socket says 172.21.15.42:37694) 2024-03-25T18:20:47.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.963+0000 7f4b0b7fe640 1 -- 172.21.15.42:0/1025727839 learned_addr learned my addr 172.21.15.42:0/1025727839 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.968+0000 7f4af37fe640 10 client.?.objecter ms_handle_connect 0x7f4b0c1430c0 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.968+0000 7f4af37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.969+0000 7f4af37fe640 10 client.?.objecter ms_handle_connect 0x7f4b0c0a98f0 2024-03-25T18:20:47.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.969+0000 7f4af37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:47.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.969+0000 7f4af37fe640 10 client.?.objecter ms_handle_connect 0x7f4b0c13f710 2024-03-25T18:20:47.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.969+0000 7f4af37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:47.976 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.975+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106612852 0 0) 0x7f4b0c0abe70 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.976 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.975+0000 7f4af37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.976 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.975+0000 7f4af37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.975+0000 7f4af37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.975+0000 7f4af37fe640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.975+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af8004eb0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.976+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068775250 0 0) 0x7f4b0c0b1880 con 0x7f4b0c13f710 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.976+0000 7f4af37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.976+0000 7f4af37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.976+0000 7f4af37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.976+0000 7f4af37fe640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:20:47.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.976+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c0abe70 con 0x7f4b0c13f710 2024-03-25T18:20:47.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.978+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640102723 0 0) 0x7f4b0c0ac150 con 0x7f4b0c1430c0 2024-03-25T18:20:47.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.978+0000 7f4af37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.978+0000 7f4af37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.978+0000 7f4af37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.978+0000 7f4af37fe640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:20:47.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.978+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c0b1880 con 0x7f4b0c1430c0 2024-03-25T18:20:47.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062251024 0 0) 0x7f4b0c0abe70 con 0x7f4b0c13f710 2024-03-25T18:20:47.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b0c0ac150 con 0x7f4b0c13f710 2024-03-25T18:20:47.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b000018d0 con 0x7f4b0c13f710 2024-03-25T18:20:47.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:47.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 10 monclient(hunting): dump: 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086849758 0 0) 0x7f4af8004eb0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b0c0abe70 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4afc001ae0 con 0x7f4b0c0a98f0 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:47.993+0000 7f4af37fe640 10 monclient(hunting): dump: 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:47.996 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.003+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763559062 0 0) 0x7f4b0c0b1880 con 0x7f4b0c1430c0 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.003+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af8004eb0 con 0x7f4b0c1430c0 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.003+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b04001b50 con 0x7f4b0c1430c0 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.003+0000 7f4af37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.003+0000 7f4af37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.003+0000 7f4af37fe640 10 monclient(hunting): dump: 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:48.005 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:48.006 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:48.006 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:48.006 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:48.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.010+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2739201307 0 0) 0x7f4af8004eb0 con 0x7f4b0c1430c0 2024-03-25T18:20:48.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 >> v1:172.21.15.120:6789/0 conn(0x7f4b0c0a98f0 legacy=0x7f4b0c1389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 >> v1:172.21.15.42:6789/0 conn(0x7f4b0c13f710 legacy=0x7f4b0c0ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:48.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 1 monclient: found mon.c 2024-03-25T18:20:48.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:48.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c0acc80 con 0x7f4b0c1430c0 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4af37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 10 monclient: _renew_subs 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c137e00 con 0x7f4b0c1430c0 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 10 monclient: _renew_subs 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c136880 con 0x7f4b0c1430c0 2024-03-25T18:20:48.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.011+0000 7f4b116c9640 1 librados: init done 2024-03-25T18:20:48.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.012+0000 7f4b116c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.012+0000 7f4b116c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.012+0000 7f4b116c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:48.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.012+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c0b3ef0 con 0x7f4b0c1430c0 2024-03-25T18:20:48.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.330+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b04002040 con 0x7f4b0c1430c0 2024-03-25T18:20:48.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.330+0000 7f4af37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:48.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.330+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b04003b30 con 0x7f4b0c1430c0 2024-03-25T18:20:48.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.330+0000 7f4af37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.330+0000 7f4af37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:48.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.330+0000 7f4af37fe640 10 monclient: dump: 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:48.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.330+0000 7f4b097fa640 4 set_mon_vals no callback set 2024-03-25T18:20:48.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.331+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f4b04003d50 con 0x7f4b0c1430c0 2024-03-25T18:20:48.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.331+0000 7f4b0affd640 10 monclient: get_auth_request con 0x7f4af806e110 auth_method 0 2024-03-25T18:20:48.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.332+0000 7f4af37fe640 10 client.4274.objecter ms_handle_connect 0x7f4af806e110 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f4b04054960 con 0x7f4b0c1430c0 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 10 client.4274.objecter ms_dispatch 0x7f4b0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b04050470 con 0x7f4b0c1430c0 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.339+0000 7f4af37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:48.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.564+0000 7f4b116c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.565 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.564+0000 7f4b116c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.565 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.564+0000 7f4b116c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:20:48.565 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.564+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b0c0627b0 con 0x7f4b0c1430c0 2024-03-25T18:20:48.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.656+0000 7f4af37fe640 1 -- 172.21.15.42:0/1025727839 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f4b040523a0 con 0x7f4b0c1430c0 2024-03-25T18:20:48.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.656+0000 7f4af37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.656+0000 7f4af37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:48.658 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:48.658 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:48.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 10 librados: watch_flush enter 2024-03-25T18:20:48.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 10 librados: watch_flush exit 2024-03-25T18:20:48.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 10 client.4274.objecter successfully canceled tick 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 >> v1:172.21.15.42:6800/3081750888 conn(0x7f4af806e110 legacy=0x7f4af80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 10 monclient: shutdown 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 >> v1:172.21.15.42:6790/0 conn(0x7f4b0c1430c0 legacy=0x7f4b0c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 shutdown_connections 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 >> 172.21.15.42:0/1025727839 conn(0x7f4b0c0b7ea0 msgr2=0x7f4b0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 shutdown_connections 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 1 -- 172.21.15.42:0/1025727839 wait complete. 2024-03-25T18:20:48.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:48.659+0000 7f4b116c9640 1 librados: shutdown 2024-03-25T18:20:48.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:55.478 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:55.598 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.597+0000 7f4b58055640 10 monclient: get_monmap_and_config 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.597+0000 7f4b58055640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.597+0000 7f4b58055640 10 monclient: monmap: 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:20:55.599004+0000 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:20:55.599004+0000 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:20:55.599 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:55.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.598+0000 7f4b58055640 1 Processor -- start 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 1 -- start start 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient: init 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient(hunting): picked mon.noname-b con 0x7f4b5006adf0 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient(hunting): picked mon.noname-c con 0x7f4b500aee10 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient(hunting): picked mon.noname-a con 0x7f4b50140860 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b500af690 con 0x7f4b50140860 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b500b0c70 con 0x7f4b500aee10 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b500b0f50 con 0x7f4b5006adf0 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.599+0000 7f4b58055640 10 monclient(hunting): authenticate will time out at 7370.826242s 2024-03-25T18:20:55.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.602+0000 7f4b555f0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f4b500aee10 0x7f4b50140150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:52006/0 (socket says 172.21.15.42:52006) 2024-03-25T18:20:55.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.602+0000 7f4b555f0640 1 -- 172.21.15.42:0/1729754466 learned_addr learned my addr 172.21.15.42:0/1729754466 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:55.613 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.612+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258289655 0 0) 0x7f4b500b0c70 con 0x7f4b500aee10 2024-03-25T18:20:55.613 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.612+0000 7f4b54def640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.613 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.612+0000 7f4b54def640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.613 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.612+0000 7f4b54def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.613 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.612+0000 7f4b54def640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:20:55.613 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.612+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c003520 con 0x7f4b500aee10 2024-03-25T18:20:55.638 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.637+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573278965 0 0) 0x7f4b500b0f50 con 0x7f4b5006adf0 2024-03-25T18:20:55.638 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.637+0000 7f4b54def640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.638 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.637+0000 7f4b54def640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.638 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.637+0000 7f4b54def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.638 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.637+0000 7f4b54def640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:20:55.638 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.637+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b500b0c70 con 0x7f4b5006adf0 2024-03-25T18:20:55.641 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1028813345 0 0) 0x7f4b3c003520 con 0x7f4b500aee10 2024-03-25T18:20:55.641 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 >> v1:172.21.15.42:6790/0 conn(0x7f4b5006adf0 legacy=0x7f4b5013d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.641 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 >> v1:172.21.15.42:6789/0 conn(0x7f4b50140860 legacy=0x7f4b5014b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:55.641 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 10 monclient: _finish_hunting 0 2024-03-25T18:20:55.641 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 1 monclient: found mon.noname-c 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5013dc40 con 0x7f4b500aee10 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 10 monclient: _finish_auth 0 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 10 monclient: _check_auth_tickets 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b40001690 con 0x7f4b500aee10 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b54def640 10 monclient: dump: 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:55.642 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b58055640 5 monclient: authenticate success, global_id 4282 2024-03-25T18:20:55.643 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.640+0000 7f4b58055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:55.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b400017f0 con 0x7f4b500aee10 2024-03-25T18:20:55.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b54def640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b54def640 1 -- 172.21.15.42:0/1729754466 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b40001c00 con 0x7f4b500aee10 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b58055640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b58055640 4 set_mon_vals no callback set 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b54def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b54def640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.648+0000 7f4b54def640 10 monclient: dump: 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:55.650 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:55.651 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:55.651 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:55.651 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.649+0000 7f4b58055640 10 monclient: shutdown 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.649+0000 7f4b58055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.649+0000 7f4b58055640 1 -- 172.21.15.42:0/1729754466 >> v1:172.21.15.120:6789/0 conn(0x7f4b500aee10 legacy=0x7f4b50140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.649+0000 7f4b58055640 1 -- 172.21.15.42:0/1729754466 shutdown_connections 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.649+0000 7f4b58055640 1 -- 172.21.15.42:0/1729754466 >> 172.21.15.42:0/1729754466 conn(0x7f4b500b4a80 msgr2=0x7f4b500b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.649+0000 7f4b58055640 1 -- 172.21.15.42:0/1729754466 shutdown_connections 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.649+0000 7f4b58055640 1 -- 172.21.15.42:0/1729754466 wait complete. 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 10 monclient: monmap: 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 1 librados: starting msgr at 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 1 librados: starting objecter 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 1 Processor -- start 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 1 -- start start 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 1 librados: setting wanted keys 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 1 librados: calling monclient init 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.650+0000 7f4b58055640 10 monclient: init 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 10 monclient(hunting): picked mon.noname-c con 0x7f4b5006adf0 addr v1:172.21.15.42:6790/0 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 10 monclient(hunting): picked mon.noname-b con 0x7f4b500aee10 addr v1:172.21.15.120:6789/0 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 10 monclient(hunting): picked mon.noname-a con 0x7f4b501355c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b501376d0 con 0x7f4b501355c0 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b50132c30 con 0x7f4b500aee10 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b501369e0 con 0x7f4b5006adf0 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.651+0000 7f4b58055640 10 monclient(hunting): authenticate will time out at 7370.878050s 2024-03-25T18:20:55.663 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.662+0000 7f4b555f0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f4b500aee10 0x7f4b50131b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:52042/0 (socket says 172.21.15.42:52042) 2024-03-25T18:20:55.663 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.662+0000 7f4b555f0640 1 -- 172.21.15.42:0/2194198167 learned_addr learned my addr 172.21.15.42:0/2194198167 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:20:55.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.664+0000 7f4b467fc640 10 client.?.objecter ms_handle_connect 0x7f4b501355c0 2024-03-25T18:20:55.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.664+0000 7f4b467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.664+0000 7f4b467fc640 10 client.?.objecter ms_handle_connect 0x7f4b500aee10 2024-03-25T18:20:55.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.664+0000 7f4b467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.671+0000 7f4b467fc640 10 client.?.objecter ms_handle_connect 0x7f4b5006adf0 2024-03-25T18:20:55.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.671+0000 7f4b467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.676+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144368105 0 0) 0x7f4b501376d0 con 0x7f4b501355c0 2024-03-25T18:20:55.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.676+0000 7f4b467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.676+0000 7f4b467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.676+0000 7f4b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.676+0000 7f4b467fc640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:20:55.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.676+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b34004eb0 con 0x7f4b501355c0 2024-03-25T18:20:55.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.684+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860548782 0 0) 0x7f4b50132c30 con 0x7f4b500aee10 2024-03-25T18:20:55.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.684+0000 7f4b467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.684+0000 7f4b467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.684+0000 7f4b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.684+0000 7f4b467fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.684+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b501376d0 con 0x7f4b500aee10 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.686+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164103400 0 0) 0x7f4b501369e0 con 0x7f4b5006adf0 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.686+0000 7f4b467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.686+0000 7f4b467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.686+0000 7f4b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.686+0000 7f4b467fc640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.686+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b50132c30 con 0x7f4b5006adf0 2024-03-25T18:20:55.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.704+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525644178 0 0) 0x7f4b34004eb0 con 0x7f4b501355c0 2024-03-25T18:20:55.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.704+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b501369e0 con 0x7f4b501355c0 2024-03-25T18:20:55.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.704+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b48001ad0 con 0x7f4b501355c0 2024-03-25T18:20:55.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.704+0000 7f4b467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.704+0000 7f4b467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.704+0000 7f4b467fc640 10 monclient(hunting): dump: 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:55.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.704+0000 7f4b467fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:55.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.705+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3995484517 0 0) 0x7f4b501376d0 con 0x7f4b500aee10 2024-03-25T18:20:55.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.705+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b34004eb0 con 0x7f4b500aee10 2024-03-25T18:20:55.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.705+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b40001b00 con 0x7f4b500aee10 2024-03-25T18:20:55.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.705+0000 7f4b467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.705+0000 7f4b467fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:55.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.705+0000 7f4b467fc640 10 monclient(hunting): dump: 2024-03-25T18:20:55.707 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.706+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1323898350 0 0) 0x7f4b50132c30 con 0x7f4b5006adf0 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.706+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b501376d0 con 0x7f4b5006adf0 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.706+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b4c0018b0 con 0x7f4b5006adf0 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.706+0000 7f4b467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.706+0000 7f4b467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.706+0000 7f4b467fc640 10 monclient(hunting): dump: 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.723+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830486701 0 0) 0x7f4b34004eb0 con 0x7f4b500aee10 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.723+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 >> v1:172.21.15.42:6790/0 conn(0x7f4b5006adf0 legacy=0x7f4b501362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.723+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 >> v1:172.21.15.42:6789/0 conn(0x7f4b501355c0 legacy=0x7f4b501333f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.723+0000 7f4b467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.723+0000 7f4b467fc640 1 monclient: found mon.b 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b50132210 con 0x7f4b500aee10 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b467fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:55.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 10 monclient: _renew_subs 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b501324c0 con 0x7f4b500aee10 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 10 monclient: _renew_subs 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b50133ce0 con 0x7f4b500aee10 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.724+0000 7f4b58055640 1 librados: init done 2024-03-25T18:20:55.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.725+0000 7f4b58055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:55.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.725+0000 7f4b58055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:55.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.725+0000 7f4b58055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:55.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:55.725+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b500a9a60 con 0x7f4b500aee10 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.027+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b400011f0 con 0x7f4b500aee10 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.027+0000 7f4b467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.027+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b40004a90 con 0x7f4b500aee10 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.027+0000 7f4b467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.027+0000 7f4b467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.027+0000 7f4b467fc640 10 monclient: dump: 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:20:56.029 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:20:56.030 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:20:56.030 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:20:56.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.027+0000 7f4b47fff640 4 set_mon_vals no callback set 2024-03-25T18:20:56.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.028+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f4b4001aa60 con 0x7f4b500aee10 2024-03-25T18:20:56.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.029+0000 7f4b55df1640 10 monclient: get_auth_request con 0x7f4b3406e160 auth_method 0 2024-03-25T18:20:56.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.029+0000 7f4b467fc640 10 client.4285.objecter ms_handle_connect 0x7f4b3406e160 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f4b40054680 con 0x7f4b500aee10 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 10 client.4285.objecter ms_dispatch 0x7f4b50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b4004ffc0 con 0x7f4b500aee10 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.033+0000 7f4b467fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:56.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.256+0000 7f4b58055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.257+0000 7f4b58055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.257+0000 7f4b58055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:20:56.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.257+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b500627b0 con 0x7f4b500aee10 2024-03-25T18:20:56.295 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.294+0000 7f4b467fc640 1 -- 172.21.15.42:0/2194198167 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f4b400520c0 con 0x7f4b500aee10 2024-03-25T18:20:56.295 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.294+0000 7f4b467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.295 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.294+0000 7f4b467fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:56.295 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:20:56.295 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:56.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.296+0000 7f4b58055640 10 librados: watch_flush enter 2024-03-25T18:20:56.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.296+0000 7f4b58055640 10 librados: watch_flush exit 2024-03-25T18:20:56.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.296+0000 7f4b58055640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T18:20:56.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.296+0000 7f4b58055640 10 client.4285.objecter successfully canceled tick 2024-03-25T18:20:56.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 >> v1:172.21.15.42:6800/3081750888 conn(0x7f4b3406e160 legacy=0x7f4b34070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 10 monclient: shutdown 2024-03-25T18:20:56.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:56.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 >> v1:172.21.15.120:6789/0 conn(0x7f4b500aee10 legacy=0x7f4b50131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 shutdown_connections 2024-03-25T18:20:56.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 >> 172.21.15.42:0/2194198167 conn(0x7f4b500b4a80 msgr2=0x7f4b5013d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:56.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 shutdown_connections 2024-03-25T18:20:56.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 1 -- 172.21.15.42:0/2194198167 wait complete. 2024-03-25T18:20:56.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:20:56.297+0000 7f4b58055640 1 librados: shutdown 2024-03-25T18:20:56.309 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:56.373 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:39.543 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.379 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:39.549 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.431 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:39.551 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.437 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:39.558 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.450 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:39.620 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.459 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:39.628 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.465 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:39.635 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.487 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:39.608 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:56.537 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:39.707 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:03.114 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:03.228 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.227+0000 7f3f42fcf640 10 monclient: get_monmap_and_config 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.227+0000 7f3f42fcf640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.228+0000 7f3f42fcf640 10 monclient: monmap: 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:03.229050+0000 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:03.229050+0000 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.229 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:03.230 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:03.230 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:03.230 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:03.230 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.228+0000 7f3f42fcf640 1 Processor -- start 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.228+0000 7f3f42fcf640 1 -- start start 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.228+0000 7f3f42fcf640 10 monclient: init 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3c0b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3c14be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:03.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0af940 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b0a70 con 0x7f3f3c0b1730 2024-03-25T18:21:03.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b0d50 con 0x7f3f3c14be90 2024-03-25T18:21:03.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.229+0000 7f3f42fcf640 10 monclient(hunting): authenticate will time out at 7378.456293s 2024-03-25T18:21:03.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.239+0000 7f3f3bfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3f3c0b1730 0x7f3f3c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:52044/0 (socket says 172.21.15.42:52044) 2024-03-25T18:21:03.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.239+0000 7f3f3bfff640 1 -- 172.21.15.42:0/3020081875 learned_addr learned my addr 172.21.15.42:0/3020081875 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:03.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.252+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435557867 0 0) 0x7f3f3c0b0a70 con 0x7f3f3c0b1730 2024-03-25T18:21:03.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.252+0000 7f3f3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.252+0000 7f3f3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.252+0000 7f3f3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.254 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.252+0000 7f3f3b7fe640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:21:03.254 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.252+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2c003520 con 0x7f3f3c0b1730 2024-03-25T18:21:03.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339019744 0 0) 0x7f3f2c003520 con 0x7f3f3c0b1730 2024-03-25T18:21:03.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 >> v1:172.21.15.42:6790/0 conn(0x7f3f3c14be90 legacy=0x7f3f3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 >> v1:172.21.15.42:6789/0 conn(0x7f3f3c0b4d60 legacy=0x7f3f3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:21:03.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0b1030 con 0x7f3f3c0b1730 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f28001690 con 0x7f3f3c0b1730 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f3b7fe640 10 monclient: dump: 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.267 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.268 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:03.268 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:03.268 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:03.268 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:03.268 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:03.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f42fcf640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:21:03.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.265+0000 7f3f42fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f280017f0 con 0x7f3f3c0b1730 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f3b7fe640 1 -- 172.21.15.42:0/3020081875 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f28001c00 con 0x7f3f3c0b1730 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f3b7fe640 10 monclient: dump: 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.279 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f42fcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:03.280 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.278+0000 7f3f42fcf640 4 set_mon_vals no callback set 2024-03-25T18:21:03.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.279+0000 7f3f42fcf640 10 monclient: shutdown 2024-03-25T18:21:03.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.279+0000 7f3f42fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:03.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.279+0000 7f3f42fcf640 1 -- 172.21.15.42:0/3020081875 >> v1:172.21.15.120:6789/0 conn(0x7f3f3c0b1730 legacy=0x7f3f3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.279+0000 7f3f42fcf640 1 -- 172.21.15.42:0/3020081875 shutdown_connections 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.279+0000 7f3f42fcf640 1 -- 172.21.15.42:0/3020081875 >> 172.21.15.42:0/3020081875 conn(0x7f3f3c0617a0 msgr2=0x7f3f3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.279+0000 7f3f42fcf640 1 -- 172.21.15.42:0/3020081875 shutdown_connections 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.279+0000 7f3f42fcf640 1 -- 172.21.15.42:0/3020081875 wait complete. 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 10 monclient: monmap: 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:03.282 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 1 librados: starting msgr at 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 1 librados: starting objecter 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 1 Processor -- start 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 1 -- start start 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 1 librados: setting wanted keys 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 1 librados: calling monclient init 2024-03-25T18:21:03.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.280+0000 7f3f42fcf640 10 monclient: init 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3c0b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c0b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3c14be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c141dc0 con 0x7f3f3c0b1730 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c140e90 con 0x7f3f3c14be90 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c141020 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.281+0000 7f3f42fcf640 10 monclient(hunting): authenticate will time out at 7378.508171s 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.282+0000 7f3f3bfff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f3f3c0b4d60 0x7f3f3c1d7ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:51696/0 (socket says 172.21.15.42:51696) 2024-03-25T18:21:03.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.282+0000 7f3f3bfff640 1 -- 172.21.15.42:0/364884297 learned_addr learned my addr 172.21.15.42:0/364884297 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:03.291 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.290+0000 7f3f397fa640 10 client.?.objecter ms_handle_connect 0x7f3f3c0b4d60 2024-03-25T18:21:03.291 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.290+0000 7f3f397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.301 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.300+0000 7f3f397fa640 10 client.?.objecter ms_handle_connect 0x7f3f3c0b1730 2024-03-25T18:21:03.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.300+0000 7f3f397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.301+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164237578 0 0) 0x7f3f3c141020 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.301+0000 7f3f397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.301+0000 7f3f397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.301+0000 7f3f397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.301+0000 7f3f397fa640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:21:03.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.301+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f20004eb0 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.305 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.304+0000 7f3f397fa640 10 client.?.objecter ms_handle_connect 0x7f3f3c14be90 2024-03-25T18:21:03.305 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.304+0000 7f3f397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.309+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545469734 0 0) 0x7f3f20004eb0 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.309+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f3c141020 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.309+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f28001bc0 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.309+0000 7f3f397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.309+0000 7f3f397fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:03.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.309+0000 7f3f397fa640 10 monclient(hunting): dump: 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:03.311 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.309+0000 7f3f397fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:03.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.310+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713336428 0 0) 0x7f3f3c141dc0 con 0x7f3f3c0b1730 2024-03-25T18:21:03.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.310+0000 7f3f397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.310+0000 7f3f397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.310+0000 7f3f397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.310+0000 7f3f397fa640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:21:03.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.310+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f20004eb0 con 0x7f3f3c0b1730 2024-03-25T18:21:03.316 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.316+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687857742 0 0) 0x7f3f3c140e90 con 0x7f3f3c14be90 2024-03-25T18:21:03.316 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.316+0000 7f3f397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.316+0000 7f3f397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.316+0000 7f3f397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.316+0000 7f3f397fa640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:21:03.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.316+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c141dc0 con 0x7f3f3c14be90 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.325+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804053169 0 0) 0x7f3f3c141dc0 con 0x7f3f3c14be90 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.325+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f3c140e90 con 0x7f3f3c14be90 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.325+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f34001a90 con 0x7f3f3c14be90 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.325+0000 7f3f397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.325+0000 7f3f397fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.325+0000 7f3f397fa640 10 monclient(hunting): dump: 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:03.326 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:03.327 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:03.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3671296316 0 0) 0x7f3f3c141020 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 >> v1:172.21.15.120:6789/0 conn(0x7f3f3c14be90 legacy=0x7f3f3c142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 >> v1:172.21.15.42:6789/0 conn(0x7f3f3c0b1730 legacy=0x7f3f3c140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 1 monclient: found mon.c 2024-03-25T18:21:03.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c1d85e0 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 10 monclient: _renew_subs 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c13b650 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 10 monclient: _renew_subs 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c142630 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.328+0000 7f3f42fcf640 1 librados: init done 2024-03-25T18:21:03.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.329+0000 7f3f42fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.329+0000 7f3f42fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.329+0000 7f3f42fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:03.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.329+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c1401f0 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.692+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f28004a90 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.692+0000 7f3f397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.692+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f28004da0 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.692+0000 7f3f397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.692+0000 7f3f397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.692+0000 7f3f397fa640 10 monclient: dump: 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:03.694 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:03.695 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:03.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.692+0000 7f3f3affd640 4 set_mon_vals no callback set 2024-03-25T18:21:03.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.693+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f3f2801ad70 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.694+0000 7f3f40d6b640 10 monclient: get_auth_request con 0x7f3f2006e020 auth_method 0 2024-03-25T18:21:03.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.694+0000 7f3f397fa640 10 client.4286.objecter ms_handle_connect 0x7f3f2006e020 2024-03-25T18:21:03.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f3f28054910 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 10 client.4286.objecter ms_dispatch 0x7f3f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:03.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:03.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:03.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-25T18:21:03.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f28050250 con 0x7f3f3c0b4d60 2024-03-25T18:21:03.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.699+0000 7f3f397fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:03.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.914+0000 7f3f42fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:03.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.914+0000 7f3f42fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:03.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.914+0000 7f3f42fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:03.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:03.914+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f3c1d8770 con 0x7f3f3c0b4d60 2024-03-25T18:21:04.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.043+0000 7f3f397fa640 1 -- 172.21.15.42:0/364884297 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f3f28001210 con 0x7f3f3c0b4d60 2024-03-25T18:21:04.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.043+0000 7f3f397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.043+0000 7f3f397fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:04.044 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:04.044 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:04.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 10 librados: watch_flush enter 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 10 librados: watch_flush exit 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 10 client.4286.objecter successfully canceled tick 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 >> v1:172.21.15.42:6800/3081750888 conn(0x7f3f2006e020 legacy=0x7f3f200704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 10 monclient: shutdown 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 >> v1:172.21.15.42:6790/0 conn(0x7f3f3c0b4d60 legacy=0x7f3f3c1d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 shutdown_connections 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 >> 172.21.15.42:0/364884297 conn(0x7f3f3c0617a0 msgr2=0x7f3f3c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 shutdown_connections 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 1 -- 172.21.15.42:0/364884297 wait complete. 2024-03-25T18:21:04.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:04.045+0000 7f3f42fcf640 1 librados: shutdown 2024-03-25T18:21:04.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:10.866 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.985+0000 7fd207b0e640 10 monclient: get_monmap_and_config 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.985+0000 7fd207b0e640 10 monclient: build_initial_monmap 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.985+0000 7fd207b0e640 10 monclient: monmap: 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:10.986324+0000 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:10.986324+0000 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:10.986 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:10.987 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:10.987 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:10.987 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:10.987 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.986+0000 7fd207b0e640 1 Processor -- start 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.986+0000 7fd207b0e640 1 -- start start 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.986+0000 7fd207b0e640 10 monclient: init 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.986+0000 7fd207b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.986+0000 7fd207b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.986+0000 7fd207b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2000a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.986+0000 7fd207b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd20013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.987+0000 7fd207b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2001430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.987+0000 7fd207b0e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd200142520 con 0x7fd2001430c0 2024-03-25T18:21:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.987+0000 7fd207b0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2000b28a0 con 0x7fd2000a98f0 2024-03-25T18:21:10.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.987+0000 7fd207b0e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2000b2a30 con 0x7fd20013f710 2024-03-25T18:21:10.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.987+0000 7fd207b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:10.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.987+0000 7fd207b0e640 10 monclient(hunting): authenticate will time out at 7386.213610s 2024-03-25T18:21:10.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.994+0000 7fd2058aa640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd2000a98f0 0x7fd2000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:42662/0 (socket says 172.21.15.42:42662) 2024-03-25T18:21:10.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:10.994+0000 7fd2058aa640 1 -- 172.21.15.42:0/3706105440 learned_addr learned my addr 172.21.15.42:0/3706105440 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:11.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.010+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350735601 0 0) 0x7fd2000b28a0 con 0x7fd2000a98f0 2024-03-25T18:21:11.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.010+0000 7fd2048a8640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.010+0000 7fd2048a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.010+0000 7fd2048a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.010+0000 7fd2048a8640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:21:11.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.010+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1ec003520 con 0x7fd2000a98f0 2024-03-25T18:21:11.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.022+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233392038 0 0) 0x7fd2000b2a30 con 0x7fd20013f710 2024-03-25T18:21:11.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.022+0000 7fd2048a8640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.022+0000 7fd2048a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.022+0000 7fd2048a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.022+0000 7fd2048a8640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:21:11.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.022+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2000b28a0 con 0x7fd20013f710 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4256189715 0 0) 0x7fd1ec003520 con 0x7fd2000a98f0 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 >> v1:172.21.15.42:6790/0 conn(0x7fd20013f710 legacy=0x7fd2001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 >> v1:172.21.15.42:6789/0 conn(0x7fd2001430c0 legacy=0x7fd20014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 10 monclient: _finish_hunting 0 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 1 monclient: found mon.noname-c 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2000b44c0 con 0x7fd2000a98f0 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 10 monclient: _finish_auth 0 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 10 monclient: _check_auth_tickets 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd1fc001690 con 0x7fd2000a98f0 2024-03-25T18:21:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd207b0e640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd207b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd2048a8640 10 monclient: dump: 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.025+0000 7fd207b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1fc0017f0 con 0x7fd2000a98f0 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd2048a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd2048a8640 1 -- 172.21.15.42:0/3706105440 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd1fc001c00 con 0x7fd2000a98f0 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd2048a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd2048a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd2048a8640 10 monclient: dump: 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:11.040 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:11.041 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:11.041 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd207b0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:11.041 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd207b0e640 4 set_mon_vals no callback set 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd207b0e640 10 monclient: shutdown 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd207b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.039+0000 7fd207b0e640 1 -- 172.21.15.42:0/3706105440 >> v1:172.21.15.120:6789/0 conn(0x7fd2000a98f0 legacy=0x7fd2000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.040+0000 7fd207b0e640 1 -- 172.21.15.42:0/3706105440 shutdown_connections 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.040+0000 7fd207b0e640 1 -- 172.21.15.42:0/3706105440 >> 172.21.15.42:0/3706105440 conn(0x7fd2000b7ea0 msgr2=0x7fd2000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.040+0000 7fd207b0e640 1 -- 172.21.15.42:0/3706105440 shutdown_connections 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.040+0000 7fd207b0e640 1 -- 172.21.15.42:0/3706105440 wait complete. 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.040+0000 7fd207b0e640 10 monclient: build_initial_monmap 2024-03-25T18:21:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.040+0000 7fd207b0e640 10 monclient: monmap: 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.041+0000 7fd207b0e640 1 librados: starting msgr at 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.041+0000 7fd207b0e640 1 librados: starting objecter 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.041+0000 7fd207b0e640 1 Processor -- start 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.041+0000 7fd207b0e640 1 -- start start 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.041+0000 7fd207b0e640 1 librados: setting wanted keys 2024-03-25T18:21:11.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.041+0000 7fd207b0e640 1 librados: calling monclient init 2024-03-25T18:21:11.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.041+0000 7fd207b0e640 10 monclient: init 2024-03-25T18:21:11.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:11.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:11.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2000a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:11.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd20013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:11.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2001430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:11.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd200135730 con 0x7fd2001430c0 2024-03-25T18:21:11.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2001358c0 con 0x7fd2000a98f0 2024-03-25T18:21:11.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd200135a50 con 0x7fd20013f710 2024-03-25T18:21:11.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:11.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.042+0000 7fd207b0e640 10 monclient(hunting): authenticate will time out at 7386.268815s 2024-03-25T18:21:11.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.050+0000 7fd2060ab640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd2001430c0 0x7fd2001d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47196/0 (socket says 172.21.15.42:47196) 2024-03-25T18:21:11.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.050+0000 7fd2060ab640 1 -- 172.21.15.42:0/1649926388 learned_addr learned my addr 172.21.15.42:0/1649926388 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:11.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.052+0000 7fd1f67fc640 10 client.?.objecter ms_handle_connect 0x7fd20013f710 2024-03-25T18:21:11.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.052+0000 7fd1f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.053+0000 7fd1f67fc640 10 client.?.objecter ms_handle_connect 0x7fd2001430c0 2024-03-25T18:21:11.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.053+0000 7fd1f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.057+0000 7fd1f67fc640 10 client.?.objecter ms_handle_connect 0x7fd2000a98f0 2024-03-25T18:21:11.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.057+0000 7fd1f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.060+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939831368 0 0) 0x7fd200135a50 con 0x7fd20013f710 2024-03-25T18:21:11.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.060+0000 7fd1f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.060+0000 7fd1f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.060+0000 7fd1f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.060+0000 7fd1f67fc640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:21:11.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.060+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e4004eb0 con 0x7fd20013f710 2024-03-25T18:21:11.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.067+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112066659 0 0) 0x7fd200135730 con 0x7fd2001430c0 2024-03-25T18:21:11.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.067+0000 7fd1f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.067+0000 7fd1f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.067+0000 7fd1f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.067+0000 7fd1f67fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:21:11.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.067+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200135a50 con 0x7fd2001430c0 2024-03-25T18:21:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.073+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443959369 0 0) 0x7fd2001358c0 con 0x7fd2000a98f0 2024-03-25T18:21:11.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.073+0000 7fd1f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.073+0000 7fd1f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.073+0000 7fd1f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.073+0000 7fd1f67fc640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:21:11.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.073+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200135730 con 0x7fd2000a98f0 2024-03-25T18:21:11.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.084+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221641219 0 0) 0x7fd1e4004eb0 con 0x7fd20013f710 2024-03-25T18:21:11.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.084+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2001358c0 con 0x7fd20013f710 2024-03-25T18:21:11.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.084+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd1f0001930 con 0x7fd20013f710 2024-03-25T18:21:11.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.084+0000 7fd1f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.084+0000 7fd1f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.084+0000 7fd1f67fc640 10 monclient(hunting): dump: 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:11.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.084+0000 7fd1f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.089+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669081326 0 0) 0x7fd200135a50 con 0x7fd2001430c0 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.089+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e4004eb0 con 0x7fd2001430c0 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.089+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd1f8001c10 con 0x7fd2001430c0 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.089+0000 7fd1f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.089+0000 7fd1f67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.089+0000 7fd1f67fc640 10 monclient(hunting): dump: 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.090 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.091 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:11.091 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:11.091 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:11.091 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:11.091 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1736310609 0 0) 0x7fd2001358c0 con 0x7fd20013f710 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 >> v1:172.21.15.120:6789/0 conn(0x7fd2000a98f0 legacy=0x7fd2001dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 >> v1:172.21.15.42:6789/0 conn(0x7fd2001430c0 legacy=0x7fd2001d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 1 monclient: found mon.c 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2001d5000 con 0x7fd20013f710 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:11.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd1f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 10 monclient: _renew_subs 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2001d52b0 con 0x7fd20013f710 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 10 monclient: _renew_subs 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20013c900 con 0x7fd20013f710 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.091+0000 7fd207b0e640 1 librados: init done 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.092+0000 7fd207b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.092+0000 7fd207b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.092+0000 7fd207b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:11.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.092+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20013cbb0 con 0x7fd20013f710 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.426+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1f0003610 con 0x7fd20013f710 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.426+0000 7fd1f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.426+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd1f0003780 con 0x7fd20013f710 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.426+0000 7fd1f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.427+0000 7fd1f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.427+0000 7fd1f67fc640 10 monclient: dump: 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:11.429 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:11.429 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:11.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.426+0000 7fd1f7fff640 4 set_mon_vals no callback set 2024-03-25T18:21:11.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.427+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fd1f0019790 con 0x7fd20013f710 2024-03-25T18:21:11.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.428+0000 7fd2058aa640 10 monclient: get_auth_request con 0x7fd1e406df20 auth_method 0 2024-03-25T18:21:11.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.428+0000 7fd1f67fc640 10 client.4292.objecter ms_handle_connect 0x7fd1e406df20 2024-03-25T18:21:11.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fd1f00534b0 con 0x7fd20013f710 2024-03-25T18:21:11.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 10 client.4292.objecter ms_dispatch 0x7fd200000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:11.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:11.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:11.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T18:21:11.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1f004edf0 con 0x7fd20013f710 2024-03-25T18:21:11.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.433+0000 7fd1f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:11.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.674+0000 7fd207b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.674+0000 7fd207b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.674+0000 7fd207b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:11.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.674+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2001371c0 con 0x7fd20013f710 2024-03-25T18:21:11.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.712+0000 7fd1f67fc640 1 -- 172.21.15.42:0/1649926388 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fd1f0050ef0 con 0x7fd20013f710 2024-03-25T18:21:11.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.712+0000 7fd1f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.712+0000 7fd1f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:11.713 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:11.713 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:11.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 10 librados: watch_flush enter 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 10 librados: watch_flush exit 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 10 client.4292.objecter successfully canceled tick 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 >> v1:172.21.15.42:6800/3081750888 conn(0x7fd1e406df20 legacy=0x7fd1e40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 10 monclient: shutdown 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 >> v1:172.21.15.42:6790/0 conn(0x7fd20013f710 legacy=0x7fd2001d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 shutdown_connections 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 >> 172.21.15.42:0/1649926388 conn(0x7fd2000b7ea0 msgr2=0x7fd20014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 shutdown_connections 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.714+0000 7fd207b0e640 1 -- 172.21.15.42:0/1649926388 wait complete. 2024-03-25T18:21:11.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:11.715+0000 7fd207b0e640 1 librados: shutdown 2024-03-25T18:21:11.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:16.392 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:59.562 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.436 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:59.557 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.438 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:59.559 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.450 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:59.619 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.479 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:01:59.600 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.483 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:01:59.652 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.500 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:59.669 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.515 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:59.684 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:16.547 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:01:59.717 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.533 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:18.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.654+0000 7fafedc0f640 10 monclient: get_monmap_and_config 2024-03-25T18:21:18.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.654+0000 7fafedc0f640 10 monclient: build_initial_monmap 2024-03-25T18:21:18.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.654+0000 7fafedc0f640 10 monclient: monmap: 2024-03-25T18:21:18.655 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:18.655530+0000 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:18.655530+0000 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:18.656 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:18.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.655+0000 7fafedc0f640 1 Processor -- start 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.655+0000 7fafedc0f640 1 -- start start 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.655+0000 7fafedc0f640 10 monclient: init 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 10 monclient(hunting): picked mon.noname-b con 0x7fafe80a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 10 monclient(hunting): picked mon.noname-c con 0x7fafe813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 10 monclient(hunting): picked mon.noname-a con 0x7fafe81430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe8142520 con 0x7fafe81430c0 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe80b28a0 con 0x7fafe813f710 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe80b2a30 con 0x7fafe80a98f0 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:18.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.656+0000 7fafedc0f640 10 monclient(hunting): authenticate will time out at 7393.882804s 2024-03-25T18:21:18.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.657+0000 7fafe6ffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fafe813f710 0x7fafe81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:41680/0 (socket says 172.21.15.42:41680) 2024-03-25T18:21:18.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.657+0000 7fafe6ffd640 1 -- 172.21.15.42:0/3495003940 learned_addr learned my addr 172.21.15.42:0/3495003940 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:18.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.676+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417998102 0 0) 0x7fafe8142520 con 0x7fafe81430c0 2024-03-25T18:21:18.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.676+0000 7fafe67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.676+0000 7fafe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.676+0000 7fafe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.676+0000 7fafe67fc640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:21:18.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.676+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd0003520 con 0x7fafe81430c0 2024-03-25T18:21:18.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2113339992 0 0) 0x7fafd0003520 con 0x7fafe81430c0 2024-03-25T18:21:18.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 >> v1:172.21.15.42:6790/0 conn(0x7fafe80a98f0 legacy=0x7fafe80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 >> v1:172.21.15.120:6789/0 conn(0x7fafe813f710 legacy=0x7fafe81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:18.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 1 monclient: found mon.noname-a 2024-03-25T18:21:18.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe80b44c0 con 0x7fafe81430c0 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fafe0001a10 con 0x7fafe81430c0 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafedc0f640 5 monclient: authenticate success, global_id 4296 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafedc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafe67fc640 10 monclient: dump: 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:18.749 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:18.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.747+0000 7fafedc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:18.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe0001b70 con 0x7fafe81430c0 2024-03-25T18:21:18.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafe67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:18.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafe67fc640 1 -- 172.21.15.42:0/3495003940 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fafe0001f80 con 0x7fafe81430c0 2024-03-25T18:21:18.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafe67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafe67fc640 10 monclient: dump: 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafedc0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:18.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.756+0000 7fafedc0f640 4 set_mon_vals no callback set 2024-03-25T18:21:18.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.757+0000 7fafedc0f640 10 monclient: shutdown 2024-03-25T18:21:18.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.757+0000 7fafedc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.757+0000 7fafedc0f640 1 -- 172.21.15.42:0/3495003940 >> v1:172.21.15.42:6789/0 conn(0x7fafe81430c0 legacy=0x7fafe814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.757+0000 7fafedc0f640 1 -- 172.21.15.42:0/3495003940 shutdown_connections 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.757+0000 7fafedc0f640 1 -- 172.21.15.42:0/3495003940 >> 172.21.15.42:0/3495003940 conn(0x7fafe80b7ea0 msgr2=0x7fafe80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.757+0000 7fafedc0f640 1 -- 172.21.15.42:0/3495003940 shutdown_connections 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.757+0000 7fafedc0f640 1 -- 172.21.15.42:0/3495003940 wait complete. 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.758+0000 7fafedc0f640 10 monclient: build_initial_monmap 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.758+0000 7fafedc0f640 10 monclient: monmap: 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:18.761 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.758+0000 7fafedc0f640 1 librados: starting msgr at 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.758+0000 7fafedc0f640 1 librados: starting objecter 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.758+0000 7fafedc0f640 1 Processor -- start 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 1 -- start start 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 1 librados: setting wanted keys 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 1 librados: calling monclient init 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient: init 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient(hunting): picked mon.noname-c con 0x7fafe80a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient(hunting): picked mon.noname-b con 0x7fafe813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient(hunting): picked mon.noname-a con 0x7fafe81430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe81d4f90 con 0x7fafe81430c0 2024-03-25T18:21:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe81d4b60 con 0x7fafe813f710 2024-03-25T18:21:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe81d4cf0 con 0x7fafe80a98f0 2024-03-25T18:21:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.759+0000 7fafedc0f640 10 monclient(hunting): authenticate will time out at 7393.986541s 2024-03-25T18:21:18.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.779+0000 7fafe7fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fafe81430c0 0x7fafe81dabb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47352/0 (socket says 172.21.15.42:47352) 2024-03-25T18:21:18.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.779+0000 7fafe7fff640 1 -- 172.21.15.42:0/1241537751 learned_addr learned my addr 172.21.15.42:0/1241537751 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:18.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.780+0000 7fafcffff640 10 client.?.objecter ms_handle_connect 0x7fafe81430c0 2024-03-25T18:21:18.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.780+0000 7fafcffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.783 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.782+0000 7fafcffff640 10 client.?.objecter ms_handle_connect 0x7fafe813f710 2024-03-25T18:21:18.783 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.782+0000 7fafcffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701693100 0 0) 0x7fafe81d4f90 con 0x7fafe81430c0 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd4004eb0 con 0x7fafe81430c0 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 10 client.?.objecter ms_handle_connect 0x7fafe80a98f0 2024-03-25T18:21:18.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.785+0000 7fafcffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.794 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.793+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030408204 0 0) 0x7fafe81d4cf0 con 0x7fafe80a98f0 2024-03-25T18:21:18.794 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.793+0000 7fafcffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.794 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.793+0000 7fafcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.794 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.793+0000 7fafcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.794 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.793+0000 7fafcffff640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:21:18.794 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.793+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe81d4f90 con 0x7fafe80a98f0 2024-03-25T18:21:18.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.798+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800171433 0 0) 0x7fafe81d4b60 con 0x7fafe813f710 2024-03-25T18:21:18.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.798+0000 7fafcffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.798+0000 7fafcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.798+0000 7fafcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.798+0000 7fafcffff640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:21:18.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.798+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe81d4cf0 con 0x7fafe813f710 2024-03-25T18:21:18.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.802+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3385219365 0 0) 0x7fafd4004eb0 con 0x7fafe81430c0 2024-03-25T18:21:18.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.802+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafe81d4b60 con 0x7fafe81430c0 2024-03-25T18:21:18.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.802+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fafe0001f40 con 0x7fafe81430c0 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.802+0000 7fafcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.802+0000 7fafcffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.802+0000 7fafcffff640 10 monclient(hunting): dump: 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:18.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.802+0000 7fafcffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:18.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.807+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3261742754 0 0) 0x7fafe81d4f90 con 0x7fafe80a98f0 2024-03-25T18:21:18.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.808+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafd4004eb0 con 0x7fafe80a98f0 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.808+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fafd80018b0 con 0x7fafe80a98f0 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.808+0000 7fafcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.808+0000 7fafcffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.808+0000 7fafcffff640 10 monclient(hunting): dump: 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:18.809 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:18.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.809+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7759816 0 0) 0x7fafe81d4cf0 con 0x7fafe813f710 2024-03-25T18:21:18.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.809+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafe81d4f90 con 0x7fafe813f710 2024-03-25T18:21:18.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.809+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fafdc001860 con 0x7fafe813f710 2024-03-25T18:21:18.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.809+0000 7fafcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.809+0000 7fafcffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.809+0000 7fafcffff640 10 monclient(hunting): dump: 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:18.811 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 354714159 0 0) 0x7fafe81d4b60 con 0x7fafe81430c0 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 >> v1:172.21.15.42:6790/0 conn(0x7fafe80a98f0 legacy=0x7fafe81d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 >> v1:172.21.15.120:6789/0 conn(0x7fafe813f710 legacy=0x7fafe80b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 1 monclient: found mon.a 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.820+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe80b08c0 con 0x7fafe81430c0 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafcffff640 10 monclient: _finish_auth 0 2024-03-25T18:21:18.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafcffff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 5 monclient: authenticate success, global_id 4299 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 10 monclient: _renew_subs 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe80b0cc0 con 0x7fafe81430c0 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 10 monclient: _renew_subs 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe81db4a0 con 0x7fafe81430c0 2024-03-25T18:21:18.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.821+0000 7fafedc0f640 1 librados: init done 2024-03-25T18:21:18.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.822+0000 7fafedc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:18.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.822+0000 7fafedc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:18.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.822+0000 7fafedc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:18.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:18.822+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe80b0b10 con 0x7fafe81430c0 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe0003c20 con 0x7fafe81430c0 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fafe0003fb0 con 0x7fafe81430c0 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafcffff640 10 monclient: dump: 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:19.131 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:19.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafe5ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:19.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.130+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fafe00041d0 con 0x7fafe81430c0 2024-03-25T18:21:19.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.131+0000 7fafe77fe640 10 monclient: get_auth_request con 0x7fafd406e0f0 auth_method 0 2024-03-25T18:21:19.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.131+0000 7fafcffff640 10 client.4299.objecter ms_handle_connect 0x7fafd406e0f0 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fafe00020a0 con 0x7fafe81430c0 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 10 client.4299.objecter ms_dispatch 0x7fafe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafe0050670 con 0x7fafe81430c0 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:19.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.137+0000 7fafcffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:19.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.389+0000 7fafedc0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:19.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.389+0000 7fafedc0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:19.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.389+0000 7fafedc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:19.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.389+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafe81da8f0 con 0x7fafe81430c0 2024-03-25T18:21:19.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.415+0000 7fafcffff640 1 -- 172.21.15.42:0/1241537751 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fafe81da8f0 con 0x7fafe81430c0 2024-03-25T18:21:19.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.415+0000 7fafcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:19.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.415+0000 7fafcffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:19.416 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:19.416 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 10 librados: watch_flush enter 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 10 librados: watch_flush exit 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 10 client.4299.objecter successfully canceled tick 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 >> v1:172.21.15.42:6800/3081750888 conn(0x7fafd406e0f0 legacy=0x7fafd40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 10 monclient: shutdown 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.417+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 >> v1:172.21.15.42:6789/0 conn(0x7fafe81430c0 legacy=0x7fafe81dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.418+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 shutdown_connections 2024-03-25T18:21:19.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.418+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 >> 172.21.15.42:0/1241537751 conn(0x7fafe80b7ea0 msgr2=0x7fafe8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:19.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.418+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 shutdown_connections 2024-03-25T18:21:19.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.418+0000 7fafedc0f640 1 -- 172.21.15.42:0/1241537751 wait complete. 2024-03-25T18:21:19.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:19.418+0000 7fafedc0f640 1 librados: shutdown 2024-03-25T18:21:19.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:26.238 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:26.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.340+0000 7f9ad8bc4640 10 monclient: get_monmap_and_config 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.340+0000 7f9ad8bc4640 10 monclient: build_initial_monmap 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.340+0000 7f9ad8bc4640 10 monclient: monmap: 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:26.340973+0000 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:26.340973+0000 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:26.341 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:26.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.341+0000 7f9ad8bc4640 1 Processor -- start 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.341+0000 7f9ad8bc4640 1 -- start start 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.341+0000 7f9ad8bc4640 10 monclient: init 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad40a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad41430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad4142520 con 0x7f9ad40a98f0 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad40b28a0 con 0x7f9ad413f710 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad40b2a30 con 0x7f9ad41430c0 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:26.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.342+0000 7f9ad8bc4640 10 monclient(hunting): authenticate will time out at 7401.568021s 2024-03-25T18:21:26.358 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.358+0000 7f9acbfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9ad413f710 0x7f9ad41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:46594/0 (socket says 172.21.15.42:46594) 2024-03-25T18:21:26.358 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.358+0000 7f9acbfff640 1 -- 172.21.15.42:0/3923400319 learned_addr learned my addr 172.21.15.42:0/3923400319 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.368+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446176114 0 0) 0x7f9ad40b28a0 con 0x7f9ad413f710 2024-03-25T18:21:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.368+0000 7f9ad1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.368+0000 7f9ad1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.368+0000 7f9ad1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.368+0000 7f9ad1d9b640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:21:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.368+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac0003520 con 0x7f9ad413f710 2024-03-25T18:21:26.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.369+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434183778 0 0) 0x7f9ad40b2a30 con 0x7f9ad41430c0 2024-03-25T18:21:26.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.369+0000 7f9ad1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.369+0000 7f9ad1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.369+0000 7f9ad1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.369+0000 7f9ad1d9b640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:21:26.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.369+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad40b28a0 con 0x7f9ad41430c0 2024-03-25T18:21:26.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667100368 0 0) 0x7f9ad40b28a0 con 0x7f9ad41430c0 2024-03-25T18:21:26.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 >> v1:172.21.15.120:6789/0 conn(0x7f9ad413f710 legacy=0x7f9ad41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 >> v1:172.21.15.42:6789/0 conn(0x7f9ad40a98f0 legacy=0x7f9ad40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40b44c0 con 0x7f9ad41430c0 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9acc0019f0 con 0x7f9ad41430c0 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad8bc4640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad8bc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:26.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad1d9b640 10 monclient: dump: 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:26.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.381+0000 7f9ad8bc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9acc001b50 con 0x7f9ad41430c0 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad1d9b640 1 -- 172.21.15.42:0/3923400319 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9acc001fa0 con 0x7f9ad41430c0 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad1d9b640 10 monclient: dump: 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.391 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad8bc4640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:26.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.391+0000 7f9ad8bc4640 4 set_mon_vals no callback set 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.392+0000 7f9ad8bc4640 10 monclient: shutdown 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.392+0000 7f9ad8bc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.392+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/3923400319 >> v1:172.21.15.42:6790/0 conn(0x7f9ad41430c0 legacy=0x7f9ad414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.392+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/3923400319 shutdown_connections 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.392+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/3923400319 >> 172.21.15.42:0/3923400319 conn(0x7f9ad40b7ea0 msgr2=0x7f9ad40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.392+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/3923400319 shutdown_connections 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.392+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/3923400319 wait complete. 2024-03-25T18:21:26.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.393+0000 7f9ad8bc4640 10 monclient: build_initial_monmap 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.393+0000 7f9ad8bc4640 10 monclient: monmap: 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:26.394 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 librados: starting msgr at 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 librados: starting objecter 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 Processor -- start 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 -- start start 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 librados: setting wanted keys 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 librados: calling monclient init 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient: init 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:26.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:26.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad4137e20 con 0x7f9ad41430c0 2024-03-25T18:21:26.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad40b1720 con 0x7f9ad413f710 2024-03-25T18:21:26.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad4138710 con 0x7f9ad40a98f0 2024-03-25T18:21:26.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:26.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.394+0000 7f9ad8bc4640 10 monclient(hunting): authenticate will time out at 7401.620446s 2024-03-25T18:21:26.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.402+0000 7f9acbfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f9ad413f710 0x7f9ad41da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:46626/0 (socket says 172.21.15.42:46626) 2024-03-25T18:21:26.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.402+0000 7f9acbfff640 1 -- 172.21.15.42:0/1201767582 learned_addr learned my addr 172.21.15.42:0/1201767582 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:26.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.404+0000 7f9acaffd640 10 client.?.objecter ms_handle_connect 0x7f9ad413f710 2024-03-25T18:21:26.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.404+0000 7f9acaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:26.415 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 10 client.?.objecter ms_handle_connect 0x7f9ad40a98f0 2024-03-25T18:21:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203305734 0 0) 0x7f9ad40b1720 con 0x7f9ad413f710 2024-03-25T18:21:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:21:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.416+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab4004eb0 con 0x7f9ad413f710 2024-03-25T18:21:26.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.424+0000 7f9acaffd640 10 client.?.objecter ms_handle_connect 0x7f9ad41430c0 2024-03-25T18:21:26.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.424+0000 7f9acaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:26.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.430+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077134564 0 0) 0x7f9ad4138710 con 0x7f9ad40a98f0 2024-03-25T18:21:26.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.430+0000 7f9acaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.430+0000 7f9acaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.430+0000 7f9acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.430+0000 7f9acaffd640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:21:26.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.430+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad40b1720 con 0x7f9ad40a98f0 2024-03-25T18:21:26.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.433+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28839901 0 0) 0x7f9ad4137e20 con 0x7f9ad41430c0 2024-03-25T18:21:26.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.433+0000 7f9acaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.433+0000 7f9acaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.433+0000 7f9acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.433+0000 7f9acaffd640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:21:26.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.433+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad4138710 con 0x7f9ad41430c0 2024-03-25T18:21:26.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.435+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3013471322 0 0) 0x7f9ab4004eb0 con 0x7f9ad413f710 2024-03-25T18:21:26.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.435+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ad4137e20 con 0x7f9ad413f710 2024-03-25T18:21:26.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.435+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9abc001950 con 0x7f9ad413f710 2024-03-25T18:21:26.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.435+0000 7f9acaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.435+0000 7f9acaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.435+0000 7f9acaffd640 10 monclient(hunting): dump: 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:26.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.435+0000 7f9acaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.445+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1446173916 0 0) 0x7f9ad4138710 con 0x7f9ad41430c0 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.445+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab4004eb0 con 0x7f9ad41430c0 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.445+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9acc001e60 con 0x7f9ad41430c0 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.445+0000 7f9acaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.445+0000 7f9acaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.445+0000 7f9acaffd640 10 monclient(hunting): dump: 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.445 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.446 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:26.446 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:26.446 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:26.446 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:26.446 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:26.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2218586848 0 0) 0x7f9ab4004eb0 con 0x7f9ad41430c0 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 >> v1:172.21.15.42:6790/0 conn(0x7f9ad40a98f0 legacy=0x7f9ad4137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 >> v1:172.21.15.120:6789/0 conn(0x7f9ad413f710 legacy=0x7f9ad41da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 1 monclient: found mon.a 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4138040 con 0x7f9ad41430c0 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:26.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9acaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 5 monclient: authenticate success, global_id 4305 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 10 monclient: _renew_subs 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad41382f0 con 0x7f9ad41430c0 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 10 monclient: _renew_subs 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad41e2fb0 con 0x7f9ad41430c0 2024-03-25T18:21:26.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.460+0000 7f9ad8bc4640 1 librados: init done 2024-03-25T18:21:26.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.461+0000 7f9ad8bc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:26.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.461+0000 7f9ad8bc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:26.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.461+0000 7f9ad8bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:26.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.461+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad40b3ef0 con 0x7f9ad41430c0 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.788+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9acc001fe0 con 0x7f9ad41430c0 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.788+0000 7f9acaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.788+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9acc003dc0 con 0x7f9ad41430c0 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.788+0000 7f9acaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.788+0000 7f9acaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.788+0000 7f9acaffd640 10 monclient: dump: 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.789 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:26.790 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.790 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:26.790 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:26.790 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:26.790 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:26.790 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:26.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.788+0000 7f9ad159a640 4 set_mon_vals no callback set 2024-03-25T18:21:26.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.789+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f9acc019d90 con 0x7f9ad41430c0 2024-03-25T18:21:26.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.790+0000 7f9ad259c640 10 monclient: get_auth_request con 0x7f9ab406dfb0 auth_method 0 2024-03-25T18:21:26.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.790+0000 7f9acaffd640 10 client.4305.objecter ms_handle_connect 0x7f9ab406dfb0 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f9acc0538b0 con 0x7f9ad41430c0 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 10 client.4305.objecter ms_dispatch 0x7f9ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9acc04f3c0 con 0x7f9ad41430c0 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:26.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:26.799+0000 7f9acaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:27.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.062+0000 7f9ad8bc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:27.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.062+0000 7f9ad8bc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:27.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.062+0000 7f9ad8bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:21:27.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.062+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ad40627b0 con 0x7f9ad41430c0 2024-03-25T18:21:27.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.105+0000 7f9acaffd640 1 -- 172.21.15.42:0/1201767582 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f9acc0512f0 con 0x7f9ad41430c0 2024-03-25T18:21:27.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.105+0000 7f9acaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:27.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.105+0000 7f9acaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:27.107 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:27.107 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:27.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.107+0000 7f9ad8bc4640 10 librados: watch_flush enter 2024-03-25T18:21:27.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.107+0000 7f9ad8bc4640 10 librados: watch_flush exit 2024-03-25T18:21:27.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.107+0000 7f9ad8bc4640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.107+0000 7f9ad8bc4640 10 client.4305.objecter successfully canceled tick 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 >> v1:172.21.15.42:6800/3081750888 conn(0x7f9ab406dfb0 legacy=0x7f9ab4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 10 monclient: shutdown 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 >> v1:172.21.15.42:6789/0 conn(0x7f9ad41430c0 legacy=0x7f9ad41daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 shutdown_connections 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 >> 172.21.15.42:0/1201767582 conn(0x7f9ad40b7ea0 msgr2=0x7f9ad4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 shutdown_connections 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 1 -- 172.21.15.42:0/1201767582 wait complete. 2024-03-25T18:21:27.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:27.108+0000 7f9ad8bc4640 1 librados: shutdown 2024-03-25T18:21:27.122 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:33.928 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:34.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.047+0000 7f5fdb89a640 10 monclient: get_monmap_and_config 2024-03-25T18:21:34.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.047+0000 7f5fdb89a640 10 monclient: build_initial_monmap 2024-03-25T18:21:34.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.047+0000 7f5fdb89a640 10 monclient: monmap: 2024-03-25T18:21:34.048 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:34.048 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:34.048 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:34.048706+0000 2024-03-25T18:21:34.049 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:34.048706+0000 2024-03-25T18:21:34.049 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:34.049 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:34.049 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:34.049 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:34.049 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:34.049 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:34.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.048+0000 7f5fdb89a640 1 Processor -- start 2024-03-25T18:21:34.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.048+0000 7f5fdb89a640 1 -- start start 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.048+0000 7f5fdb89a640 10 monclient: init 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd4061830 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd4071dc0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd413a230 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd4073100 con 0x7f5fd4071dc0 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd413dc20 con 0x7f5fd413a230 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd406ee00 con 0x7f5fd4061830 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:34.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.049+0000 7f5fdb89a640 10 monclient(hunting): authenticate will time out at 7409.276014s 2024-03-25T18:21:34.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.053+0000 7f5fd9e37640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f5fd413a230 0x7f5fd4140a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:38096/0 (socket says 172.21.15.42:38096) 2024-03-25T18:21:34.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.053+0000 7f5fd9e37640 1 -- 172.21.15.42:0/2535460758 learned_addr learned my addr 172.21.15.42:0/2535460758 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:34.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.065+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421164314 0 0) 0x7f5fd413dc20 con 0x7f5fd413a230 2024-03-25T18:21:34.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.065+0000 7f5fcbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.065+0000 7f5fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.065+0000 7f5fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.065+0000 7f5fcbfff640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:21:34.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.065+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc0003520 con 0x7f5fd413a230 2024-03-25T18:21:34.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.081+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333392433 0 0) 0x7f5fd406ee00 con 0x7f5fd4061830 2024-03-25T18:21:34.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.081+0000 7f5fcbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.081+0000 7f5fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.081+0000 7f5fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.081+0000 7f5fcbfff640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:21:34.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.081+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd413dc20 con 0x7f5fd4061830 2024-03-25T18:21:34.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1431864543 0 0) 0x7f5fc0003520 con 0x7f5fd413a230 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 >> v1:172.21.15.42:6790/0 conn(0x7f5fd4061830 legacy=0x7f5fd4139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 >> v1:172.21.15.42:6789/0 conn(0x7f5fd4071dc0 legacy=0x7f5fd4140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 1 monclient: found mon.noname-c 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd414fa40 con 0x7f5fd413a230 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 10 monclient: _finish_auth 0 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f5fcc001a10 con 0x7f5fd413a230 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fcbfff640 10 monclient: dump: 2024-03-25T18:21:34.084 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fdb89a640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:21:34.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.082+0000 7f5fdb89a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:34.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fcc001b70 con 0x7f5fd413a230 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fcbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fcbfff640 1 -- 172.21.15.42:0/2535460758 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f5fcc001f00 con 0x7f5fd413a230 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fdb89a640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fdb89a640 4 set_mon_vals no callback set 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fcbfff640 10 monclient: dump: 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.094 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:34.095 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:34.095 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:34.095 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:34.095 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fdb89a640 10 monclient: shutdown 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fdb89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fdb89a640 1 -- 172.21.15.42:0/2535460758 >> v1:172.21.15.120:6789/0 conn(0x7f5fd413a230 legacy=0x7f5fd4140a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fdb89a640 1 -- 172.21.15.42:0/2535460758 shutdown_connections 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.093+0000 7f5fdb89a640 1 -- 172.21.15.42:0/2535460758 >> 172.21.15.42:0/2535460758 conn(0x7f5fd40b1600 msgr2=0x7f5fd406ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.094+0000 7f5fdb89a640 1 -- 172.21.15.42:0/2535460758 shutdown_connections 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.094+0000 7f5fdb89a640 1 -- 172.21.15.42:0/2535460758 wait complete. 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.094+0000 7f5fdb89a640 10 monclient: build_initial_monmap 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.094+0000 7f5fdb89a640 10 monclient: monmap: 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:34.097 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.095+0000 7f5fdb89a640 1 librados: starting msgr at 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.095+0000 7f5fdb89a640 1 librados: starting objecter 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.095+0000 7f5fdb89a640 1 Processor -- start 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.095+0000 7f5fdb89a640 1 -- start start 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.095+0000 7f5fdb89a640 1 librados: setting wanted keys 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.095+0000 7f5fdb89a640 1 librados: calling monclient init 2024-03-25T18:21:34.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.095+0000 7f5fdb89a640 10 monclient: init 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd4061830 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd4071dc0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd413a230 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd406fc80 con 0x7f5fd4071dc0 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd4070c70 con 0x7f5fd4061830 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd4070ee0 con 0x7f5fd413a230 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:34.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.096+0000 7f5fdb89a640 10 monclient(hunting): authenticate will time out at 7409.322677s 2024-03-25T18:21:34.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.105+0000 7f5fd9636640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f5fd4061830 0x7f5fd41345f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:38116/0 (socket says 172.21.15.42:38116) 2024-03-25T18:21:34.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.105+0000 7f5fd9636640 1 -- 172.21.15.42:0/880513629 learned_addr learned my addr 172.21.15.42:0/880513629 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:34.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.107+0000 7f5fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f5fd4061830 2024-03-25T18:21:34.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.107+0000 7f5fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:34.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.112+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028149985 0 0) 0x7f5fd4070c70 con 0x7f5fd4061830 2024-03-25T18:21:34.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.112+0000 7f5fc9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.112+0000 7f5fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.112+0000 7f5fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.112+0000 7f5fc9ffb640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:21:34.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.112+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb8004eb0 con 0x7f5fd4061830 2024-03-25T18:21:34.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.114+0000 7f5fd8e35640 1 -- 172.21.15.42:0/880513629 >> v1:172.21.15.42:6789/0 conn(0x7f5fd4071dc0 legacy=0x7f5fd4143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:21:34.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.114+0000 7f5fd8e35640 1 -- 172.21.15.42:0/880513629 >> v1:172.21.15.42:6789/0 conn(0x7f5fd4071dc0 legacy=0x7f5fd4143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:21:34.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.114+0000 7f5fd8e35640 1 --1- 172.21.15.42:0/880513629 >> v1:172.21.15.42:6789/0 conn(0x7f5fd4071dc0 0x7f5fd4143a60 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T18:21:34.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.114+0000 7f5fd8e35640 1 --1- 172.21.15.42:0/880513629 >> v1:172.21.15.42:6789/0 conn(0x7f5fd4071dc0 0x7f5fd4143a60 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:21:34.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.114+0000 7f5fc9ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.42:6789/0 2024-03-25T18:21:34.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.122+0000 7f5fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f5fd413a230 2024-03-25T18:21:34.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.122+0000 7f5fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:34.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.131+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2922358408 0 0) 0x7f5fb8004eb0 con 0x7f5fd4061830 2024-03-25T18:21:34.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.131+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fd4070c70 con 0x7f5fd4061830 2024-03-25T18:21:34.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.131+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f5fd0001a90 con 0x7f5fd4061830 2024-03-25T18:21:34.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.131+0000 7f5fc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.131+0000 7f5fc9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:34.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.131+0000 7f5fc9ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:34.132 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:34.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.131+0000 7f5fc9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:34.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.134+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628126577 0 0) 0x7f5fd4070ee0 con 0x7f5fd413a230 2024-03-25T18:21:34.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.134+0000 7f5fc9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.134+0000 7f5fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.134+0000 7f5fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.134+0000 7f5fc9ffb640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:21:34.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.134+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb8004eb0 con 0x7f5fd413a230 2024-03-25T18:21:34.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3232491704 0 0) 0x7f5fd4070c70 con 0x7f5fd4061830 2024-03-25T18:21:34.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 >> v1:172.21.15.42:6790/0 conn(0x7f5fd413a230 legacy=0x7f5fd41e2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 >> v1:172.21.15.42:6789/0 conn(0x7f5fd4071dc0 legacy=0x7f5fd4143a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:21:34.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:34.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 1 monclient: found mon.b 2024-03-25T18:21:34.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd40711c0 con 0x7f5fd4061830 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 10 monclient: _renew_subs 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:34.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd40714a0 con 0x7f5fd4061830 2024-03-25T18:21:34.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:34.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 10 monclient: _renew_subs 2024-03-25T18:21:34.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:34.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd413d3d0 con 0x7f5fd4061830 2024-03-25T18:21:34.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.154+0000 7f5fdb89a640 1 librados: init done 2024-03-25T18:21:34.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.155+0000 7f5fdb89a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:34.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.155+0000 7f5fdb89a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:34.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.155+0000 7f5fdb89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:34.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.155+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd406fe30 con 0x7f5fd4061830 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.425+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd0001d10 con 0x7f5fd4061830 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.426+0000 7f5fc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.426+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f5fd00038c0 con 0x7f5fd4061830 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.426+0000 7f5fc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.426+0000 7f5fcb7fe640 4 set_mon_vals no callback set 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.426+0000 7f5fc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.426+0000 7f5fc9ffb640 10 monclient: dump: 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:34.427 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.428 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:34.428 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:34.428 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:34.428 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:34.428 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:34.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.427+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f5fd0003aa0 con 0x7f5fd4061830 2024-03-25T18:21:34.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.427+0000 7f5fd8e35640 10 monclient: get_auth_request con 0x7f5fb806dd30 auth_method 0 2024-03-25T18:21:34.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.428+0000 7f5fc9ffb640 10 client.4312.objecter ms_handle_connect 0x7f5fb806dd30 2024-03-25T18:21:34.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.431+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f5fd0054750 con 0x7f5fd4061830 2024-03-25T18:21:34.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.431+0000 7f5fc9ffb640 10 client.4312.objecter ms_dispatch 0x7f5fd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:34.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.431+0000 7f5fc9ffb640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:34.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.431+0000 7f5fc9ffb640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:34.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.431+0000 7f5fc9ffb640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T18:21:34.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.432+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fd0050260 con 0x7f5fd4061830 2024-03-25T18:21:34.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.432+0000 7f5fc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:34.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.432+0000 7f5fc9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:34.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.682+0000 7f5fdb89a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:34.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.682+0000 7f5fdb89a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:34.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.682+0000 7f5fdb89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:34.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.682+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fd40627b0 con 0x7f5fd4061830 2024-03-25T18:21:34.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.717+0000 7f5fc9ffb640 1 -- 172.21.15.42:0/880513629 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f5fd0052190 con 0x7f5fd4061830 2024-03-25T18:21:34.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.717+0000 7f5fc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:34.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.717+0000 7f5fc9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:34.718 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:34.718 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:34.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 10 librados: watch_flush enter 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 10 librados: watch_flush exit 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 10 client.4312.objecter successfully canceled tick 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 >> v1:172.21.15.42:6800/3081750888 conn(0x7f5fb806dd30 legacy=0x7f5fb80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 10 monclient: shutdown 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 >> v1:172.21.15.120:6789/0 conn(0x7f5fd4061830 legacy=0x7f5fd41345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 shutdown_connections 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.719+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 >> 172.21.15.42:0/880513629 conn(0x7f5fd40b1600 msgr2=0x7f5fd413e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.720+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 shutdown_connections 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.720+0000 7f5fdb89a640 1 -- 172.21.15.42:0/880513629 wait complete. 2024-03-25T18:21:34.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:34.720+0000 7f5fdb89a640 1 librados: shutdown 2024-03-25T18:21:34.733 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:36.503 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:02:19.672 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.523 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:02:19.692 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.552 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:02:19.673 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.557 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:02:19.726 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.562 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:02:19.683 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.564 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:02:19.734 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.577 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:02:19.747 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.584 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:02:19.704 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:36.611 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:02:19.781 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:41.540 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:41.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.653+0000 7fd468633640 10 monclient: get_monmap_and_config 2024-03-25T18:21:41.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.653+0000 7fd468633640 10 monclient: build_initial_monmap 2024-03-25T18:21:41.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.653+0000 7fd468633640 10 monclient: monmap: 2024-03-25T18:21:41.654 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:41.654362+0000 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:41.654362+0000 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:41.655 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:41.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 1 Processor -- start 2024-03-25T18:21:41.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 1 -- start start 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient: init 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient(hunting): picked mon.noname-a con 0x7fd4600a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient(hunting): picked mon.noname-c con 0x7fd46013f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient(hunting): picked mon.noname-b con 0x7fd4601430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd460142520 con 0x7fd4600a98f0 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600b28a0 con 0x7fd46013f710 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600b2a30 con 0x7fd4601430c0 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:41.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.654+0000 7fd468633640 10 monclient(hunting): authenticate will time out at 7416.881490s 2024-03-25T18:21:41.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.663+0000 7fd465bce640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd46013f710 0x7fd4601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:33068/0 (socket says 172.21.15.42:33068) 2024-03-25T18:21:41.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.663+0000 7fd465bce640 1 -- 172.21.15.42:0/3808732436 learned_addr learned my addr 172.21.15.42:0/3808732436 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:41.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.670+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646891491 0 0) 0x7fd4600b28a0 con 0x7fd46013f710 2024-03-25T18:21:41.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.671+0000 7fd4653cd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.671+0000 7fd4653cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.671+0000 7fd4653cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.671+0000 7fd4653cd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:21:41.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.671+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd44c003520 con 0x7fd46013f710 2024-03-25T18:21:41.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.676+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592169945 0 0) 0x7fd4600b2a30 con 0x7fd4601430c0 2024-03-25T18:21:41.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.676+0000 7fd4653cd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.676+0000 7fd4653cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.676+0000 7fd4653cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.676+0000 7fd4653cd640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:21:41.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.676+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4600b28a0 con 0x7fd4601430c0 2024-03-25T18:21:41.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3908488921 0 0) 0x7fd44c003520 con 0x7fd46013f710 2024-03-25T18:21:41.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 >> v1:172.21.15.42:6790/0 conn(0x7fd4601430c0 legacy=0x7fd46014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 >> v1:172.21.15.42:6789/0 conn(0x7fd4600a98f0 legacy=0x7fd4600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 1 monclient: found mon.noname-c 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4600b44c0 con 0x7fd46013f710 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 10 monclient: _finish_auth 0 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:41.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd450001690 con 0x7fd46013f710 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd468633640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd468633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd4653cd640 10 monclient: dump: 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.685 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:41.686 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:41.686 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:41.686 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:41.686 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:41.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.682+0000 7fd468633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4500017f0 con 0x7fd46013f710 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd4653cd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd4653cd640 1 -- 172.21.15.42:0/3808732436 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd450001c00 con 0x7fd46013f710 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd468633640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd468633640 4 set_mon_vals no callback set 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd4653cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd4653cd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd4653cd640 10 monclient: dump: 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:41.700 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:41.701 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:41.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd468633640 10 monclient: shutdown 2024-03-25T18:21:41.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd468633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:41.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd468633640 1 -- 172.21.15.42:0/3808732436 >> v1:172.21.15.120:6789/0 conn(0x7fd46013f710 legacy=0x7fd4601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd468633640 1 -- 172.21.15.42:0/3808732436 shutdown_connections 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.699+0000 7fd468633640 1 -- 172.21.15.42:0/3808732436 >> 172.21.15.42:0/3808732436 conn(0x7fd4600b7ea0 msgr2=0x7fd4600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.700+0000 7fd468633640 1 -- 172.21.15.42:0/3808732436 shutdown_connections 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.700+0000 7fd468633640 1 -- 172.21.15.42:0/3808732436 wait complete. 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.700+0000 7fd468633640 10 monclient: build_initial_monmap 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 10 monclient: monmap: 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:41.704 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 1 librados: starting msgr at 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 1 librados: starting objecter 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 1 Processor -- start 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 1 -- start start 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 1 librados: setting wanted keys 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 1 librados: calling monclient init 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.701+0000 7fd468633640 10 monclient: init 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:41.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 10 monclient(hunting): picked mon.noname-a con 0x7fd4600a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 10 monclient(hunting): picked mon.noname-b con 0x7fd46013f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 10 monclient(hunting): picked mon.noname-c con 0x7fd4601430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600b7890 con 0x7fd4600a98f0 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4601370c0 con 0x7fd46013f710 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4601357c0 con 0x7fd4601430c0 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.702+0000 7fd468633640 10 monclient(hunting): authenticate will time out at 7416.928775s 2024-03-25T18:21:41.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.705+0000 7fd466bd0640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fd4601430c0 0x7fd4601e2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:34394/0 (socket says 172.21.15.42:34394) 2024-03-25T18:21:41.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.705+0000 7fd466bd0640 1 -- 172.21.15.42:0/3104328954 learned_addr learned my addr 172.21.15.42:0/3104328954 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:41.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.706+0000 7fd456ffd640 10 client.?.objecter ms_handle_connect 0x7fd4601430c0 2024-03-25T18:21:41.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.706+0000 7fd456ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:41.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.723+0000 7fd456ffd640 10 client.?.objecter ms_handle_connect 0x7fd46013f710 2024-03-25T18:21:41.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.723+0000 7fd456ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:41.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.727+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353583726 0 0) 0x7fd4601357c0 con 0x7fd4601430c0 2024-03-25T18:21:41.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.727+0000 7fd456ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.727+0000 7fd456ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.727+0000 7fd456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.727+0000 7fd456ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:21:41.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.727+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd444004eb0 con 0x7fd4601430c0 2024-03-25T18:21:41.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.731+0000 7fd456ffd640 10 client.?.objecter ms_handle_connect 0x7fd4600a98f0 2024-03-25T18:21:41.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.731+0000 7fd456ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:41.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.732+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839928694 0 0) 0x7fd4601370c0 con 0x7fd46013f710 2024-03-25T18:21:41.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.732+0000 7fd456ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.732+0000 7fd456ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.732+0000 7fd456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.732+0000 7fd456ffd640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:21:41.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.732+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4601357c0 con 0x7fd46013f710 2024-03-25T18:21:41.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.740+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000555313 0 0) 0x7fd4600b7890 con 0x7fd4600a98f0 2024-03-25T18:21:41.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.740+0000 7fd456ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.740+0000 7fd456ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.740+0000 7fd456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.740+0000 7fd456ffd640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:21:41.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.740+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4601370c0 con 0x7fd4600a98f0 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2851488128 0 0) 0x7fd444004eb0 con 0x7fd4601430c0 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4600b7890 con 0x7fd4601430c0 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd458001c90 con 0x7fd4601430c0 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:41.746 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188102394 0 0) 0x7fd4601357c0 con 0x7fd46013f710 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd444004eb0 con 0x7fd46013f710 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd450001b00 con 0x7fd46013f710 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.745+0000 7fd456ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:41.748 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:41.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 194038991 0 0) 0x7fd4600b7890 con 0x7fd4601430c0 2024-03-25T18:21:41.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 >> v1:172.21.15.120:6789/0 conn(0x7fd46013f710 legacy=0x7fd4601dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 >> v1:172.21.15.42:6789/0 conn(0x7fd4600a98f0 legacy=0x7fd460130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 1 monclient: found mon.c 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd460135950 con 0x7fd4601430c0 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd456ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 10 monclient: _renew_subs 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd460131660 con 0x7fd4601430c0 2024-03-25T18:21:41.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:41.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 10 monclient: _renew_subs 2024-03-25T18:21:41.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:41.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46013b660 con 0x7fd4601430c0 2024-03-25T18:21:41.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.750+0000 7fd468633640 1 librados: init done 2024-03-25T18:21:41.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.751+0000 7fd468633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:41.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.751+0000 7fd468633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:41.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.751+0000 7fd468633640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:41.754 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:41.751+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd460131a00 con 0x7fd4601430c0 2024-03-25T18:21:42.055 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.054+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd458003990 con 0x7fd4601430c0 2024-03-25T18:21:42.055 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.054+0000 7fd456ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:42.055 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.054+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd458003b00 con 0x7fd4601430c0 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.054+0000 7fd456ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.054+0000 7fd456ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.054+0000 7fd456ffd640 10 monclient: dump: 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:42.056 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:42.057 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:42.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.054+0000 7fd464bcc640 4 set_mon_vals no callback set 2024-03-25T18:21:42.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.055+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fd458003d20 con 0x7fd4601430c0 2024-03-25T18:21:42.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.056+0000 7fd4663cf640 10 monclient: get_auth_request con 0x7fd44406df70 auth_method 0 2024-03-25T18:21:42.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.056+0000 7fd456ffd640 10 client.4313.objecter ms_handle_connect 0x7fd44406df70 2024-03-25T18:21:42.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fd4580548b0 con 0x7fd4601430c0 2024-03-25T18:21:42.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 10 client.4313.objecter ms_dispatch 0x7fd460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:42.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:42.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:42.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T18:21:42.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4580503c0 con 0x7fd4601430c0 2024-03-25T18:21:42.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:42.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.061+0000 7fd456ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:42.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.296+0000 7fd468633640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:42.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.296+0000 7fd468633640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:42.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.296+0000 7fd468633640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:42.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.296+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd460135ae0 con 0x7fd4601430c0 2024-03-25T18:21:42.338 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.337+0000 7fd456ffd640 1 -- 172.21.15.42:0/3104328954 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fd4580522f0 con 0x7fd4601430c0 2024-03-25T18:21:42.338 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.337+0000 7fd456ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:42.338 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.337+0000 7fd456ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:42.338 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:42.339 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 10 librados: watch_flush enter 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 10 librados: watch_flush exit 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 10 client.4313.objecter successfully canceled tick 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 >> v1:172.21.15.42:6800/3081750888 conn(0x7fd44406df70 legacy=0x7fd444070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 10 monclient: shutdown 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:42.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 >> v1:172.21.15.42:6790/0 conn(0x7fd4601430c0 legacy=0x7fd4601e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 shutdown_connections 2024-03-25T18:21:42.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.339+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 >> 172.21.15.42:0/3104328954 conn(0x7fd4600b7ea0 msgr2=0x7fd460074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:42.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.340+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 shutdown_connections 2024-03-25T18:21:42.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.340+0000 7fd468633640 1 -- 172.21.15.42:0/3104328954 wait complete. 2024-03-25T18:21:42.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:42.340+0000 7fd468633640 1 librados: shutdown 2024-03-25T18:21:42.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:49.157 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.271+0000 7ff1a1043640 10 monclient: get_monmap_and_config 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.271+0000 7ff1a1043640 10 monclient: build_initial_monmap 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.271+0000 7ff1a1043640 10 monclient: monmap: 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:49.272401+0000 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:49.272401+0000 2024-03-25T18:21:49.272 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:49.273 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:49.273 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:49.273 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:49.273 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:49.273 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 1 Processor -- start 2024-03-25T18:21:49.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 1 -- start start 2024-03-25T18:21:49.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient: init 2024-03-25T18:21:49.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:49.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:49.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient(hunting): picked mon.noname-b con 0x7ff19c0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:49.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient(hunting): picked mon.noname-a con 0x7ff19c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:49.275 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:49.275 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c142520 con 0x7ff19c13f710 2024-03-25T18:21:49.275 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0b28a0 con 0x7ff19c1430c0 2024-03-25T18:21:49.275 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0b2a30 con 0x7ff19c0a98f0 2024-03-25T18:21:49.275 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:49.275 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.272+0000 7ff1a1043640 10 monclient(hunting): authenticate will time out at 7424.499525s 2024-03-25T18:21:49.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.282+0000 7ff19b59e640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff19c1430c0 0x7ff19c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:50160/0 (socket says 172.21.15.42:50160) 2024-03-25T18:21:49.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.283+0000 7ff19b59e640 1 -- 172.21.15.42:0/25548260 learned_addr learned my addr 172.21.15.42:0/25548260 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:49.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.283+0000 7ff19b59e640 1 -- 172.21.15.42:0/25548260 >> v1:172.21.15.120:6789/0 conn(0x7ff19c1430c0 legacy=0x7ff19c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T18:21:49.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.283+0000 7ff19b59e640 1 -- 172.21.15.42:0/25548260 >> v1:172.21.15.120:6789/0 conn(0x7ff19c1430c0 legacy=0x7ff19c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:21:49.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.283+0000 7ff19b59e640 1 --1- 172.21.15.42:0/25548260 >> v1:172.21.15.120:6789/0 conn(0x7ff19c1430c0 0x7ff19c14dc30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:21:49.306 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.305+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670074811 0 0) 0x7ff19c142520 con 0x7ff19c13f710 2024-03-25T18:21:49.306 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.305+0000 7ff19a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.305+0000 7ff19a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.305+0000 7ff19a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.305+0000 7ff19a59c640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:21:49.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.305+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff188003520 con 0x7ff19c13f710 2024-03-25T18:21:49.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.311+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016979339 0 0) 0x7ff19c0b2a30 con 0x7ff19c0a98f0 2024-03-25T18:21:49.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.311+0000 7ff19a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.311+0000 7ff19a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.311+0000 7ff19a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.311+0000 7ff19a59c640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:21:49.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.311+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c142520 con 0x7ff19c0a98f0 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681973916 0 0) 0x7ff188003520 con 0x7ff19c13f710 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 >> v1:172.21.15.42:6790/0 conn(0x7ff19c0a98f0 legacy=0x7ff19c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 >> v1:172.21.15.120:6789/0 conn(0x7ff19c1430c0 legacy=0x7ff19c14dc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 1 monclient: found mon.noname-a 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c0b44c0 con 0x7ff19c13f710 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 10 monclient: _finish_auth 0 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff184001690 con 0x7ff19c13f710 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff1a1043640 5 monclient: authenticate success, global_id 4317 2024-03-25T18:21:49.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff1a1043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff19a59c640 10 monclient: dump: 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.324+0000 7ff1a1043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1840017f0 con 0x7ff19c13f710 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff19a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff19a59c640 1 -- 172.21.15.42:0/25548260 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff184001c40 con 0x7ff19c13f710 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff1a1043640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff1a1043640 4 set_mon_vals no callback set 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff19a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff19a59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff19a59c640 10 monclient: dump: 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.345 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.346 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.346 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:49.346 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:49.346 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:49.346 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:49.346 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff1a1043640 10 monclient: shutdown 2024-03-25T18:21:49.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff1a1043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:49.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff1a1043640 1 -- 172.21.15.42:0/25548260 >> v1:172.21.15.42:6789/0 conn(0x7ff19c13f710 legacy=0x7ff19c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff1a1043640 1 -- 172.21.15.42:0/25548260 shutdown_connections 2024-03-25T18:21:49.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.344+0000 7ff1a1043640 1 -- 172.21.15.42:0/25548260 >> 172.21.15.42:0/25548260 conn(0x7ff19c0b7ea0 msgr2=0x7ff19c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:49.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.345+0000 7ff1a1043640 1 -- 172.21.15.42:0/25548260 shutdown_connections 2024-03-25T18:21:49.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.345+0000 7ff1a1043640 1 -- 172.21.15.42:0/25548260 wait complete. 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.345+0000 7ff1a1043640 10 monclient: build_initial_monmap 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.345+0000 7ff1a1043640 10 monclient: monmap: 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.346+0000 7ff1a1043640 1 librados: starting msgr at 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.346+0000 7ff1a1043640 1 librados: starting objecter 2024-03-25T18:21:49.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.346+0000 7ff1a1043640 1 Processor -- start 2024-03-25T18:21:49.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.346+0000 7ff1a1043640 1 -- start start 2024-03-25T18:21:49.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.346+0000 7ff1a1043640 1 librados: setting wanted keys 2024-03-25T18:21:49.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.346+0000 7ff1a1043640 1 librados: calling monclient init 2024-03-25T18:21:49.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.346+0000 7ff1a1043640 10 monclient: init 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 10 monclient(hunting): picked mon.noname-a con 0x7ff19c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 10 monclient(hunting): picked mon.noname-b con 0x7ff19c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0b7430 con 0x7ff19c13f710 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c13b6d0 con 0x7ff19c1430c0 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0b7180 con 0x7ff19c0a98f0 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:49.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.347+0000 7ff1a1043640 10 monclient(hunting): authenticate will time out at 7424.573999s 2024-03-25T18:21:49.355 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.355+0000 7ff19b59e640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff19c1430c0 0x7ff19c1e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:50208/0 (socket says 172.21.15.42:50208) 2024-03-25T18:21:49.355 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.355+0000 7ff19b59e640 1 -- 172.21.15.42:0/3115560405 learned_addr learned my addr 172.21.15.42:0/3115560405 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:49.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.364+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff19c1430c0 2024-03-25T18:21:49.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.364+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:49.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.369+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff19c0a98f0 2024-03-25T18:21:49.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.369+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:49.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.370+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff19c13f710 2024-03-25T18:21:49.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.370+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:49.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.378+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639271786 0 0) 0x7ff19c13b6d0 con 0x7ff19c1430c0 2024-03-25T18:21:49.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.378+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.378+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.378+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.378+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:21:49.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.378+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff17c004eb0 con 0x7ff19c1430c0 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527457927 0 0) 0x7ff19c0b7430 con 0x7ff19c13f710 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c13b6d0 con 0x7ff19c13f710 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122801066 0 0) 0x7ff19c0b7180 con 0x7ff19c0a98f0 2024-03-25T18:21:49.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:21:49.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.390+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c0b7430 con 0x7ff19c0a98f0 2024-03-25T18:21:49.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.395+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441997846 0 0) 0x7ff17c004eb0 con 0x7ff19c1430c0 2024-03-25T18:21:49.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.395+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff19c0b7180 con 0x7ff19c1430c0 2024-03-25T18:21:49.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.395+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff194001890 con 0x7ff19c1430c0 2024-03-25T18:21:49.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.395+0000 7ff1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.395+0000 7ff1937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.395+0000 7ff1937fe640 10 monclient(hunting): dump: 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.395+0000 7ff1937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:49.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.412+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2373951787 0 0) 0x7ff19c0b7430 con 0x7ff19c0a98f0 2024-03-25T18:21:49.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.413+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff17c004eb0 con 0x7ff19c0a98f0 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.413+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff18c0016b0 con 0x7ff19c0a98f0 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.413+0000 7ff1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.413+0000 7ff1937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.413+0000 7ff1937fe640 10 monclient(hunting): dump: 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:49.414 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:49.415 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.426+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2669448320 0 0) 0x7ff19c13b6d0 con 0x7ff19c13f710 2024-03-25T18:21:49.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.426+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff19c0b7430 con 0x7ff19c13f710 2024-03-25T18:21:49.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.426+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff184001b40 con 0x7ff19c13f710 2024-03-25T18:21:49.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.426+0000 7ff1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.426+0000 7ff1937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:49.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.426+0000 7ff1937fe640 10 monclient(hunting): dump: 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:49.428 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4015306134 0 0) 0x7ff19c0b7180 con 0x7ff19c1430c0 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 >> v1:172.21.15.42:6790/0 conn(0x7ff19c0a98f0 legacy=0x7ff19c12f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 >> v1:172.21.15.42:6789/0 conn(0x7ff19c13f710 legacy=0x7ff19c139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 1 monclient: found mon.b 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c130260 con 0x7ff19c1430c0 2024-03-25T18:21:49.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.432+0000 7ff1937fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 10 monclient: _renew_subs 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c1397a0 con 0x7ff19c1430c0 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 10 monclient: _renew_subs 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c13a610 con 0x7ff19c1430c0 2024-03-25T18:21:49.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.433+0000 7ff1a1043640 1 librados: init done 2024-03-25T18:21:49.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.434+0000 7ff1a1043640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:49.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.434+0000 7ff1a1043640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:49.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.434+0000 7ff1a1043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:49.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.434+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c13b4f0 con 0x7ff19c1430c0 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.707+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff194001d80 con 0x7ff19c1430c0 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.707+0000 7ff1937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.707+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff194004e00 con 0x7ff19c1430c0 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.707+0000 7ff1937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.707+0000 7ff1937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.707+0000 7ff1937fe640 10 monclient: dump: 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:49.709 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.710 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:49.710 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:49.710 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:49.710 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:49.710 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:49.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.707+0000 7ff199d9b640 4 set_mon_vals no callback set 2024-03-25T18:21:49.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.708+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff19401add0 con 0x7ff19c1430c0 2024-03-25T18:21:49.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.709+0000 7ff19ad9d640 10 monclient: get_auth_request con 0x7ff17c06e0b0 auth_method 0 2024-03-25T18:21:49.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.709+0000 7ff1937fe640 10 client.4321.objecter ms_handle_connect 0x7ff17c06e0b0 2024-03-25T18:21:49.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.712+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7ff194054a90 con 0x7ff19c1430c0 2024-03-25T18:21:49.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.712+0000 7ff1937fe640 10 client.4321.objecter ms_dispatch 0x7ff19c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:49.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.712+0000 7ff1937fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:49.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.712+0000 7ff1937fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:49.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.712+0000 7ff1937fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T18:21:49.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.717+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1940505a0 con 0x7ff19c1430c0 2024-03-25T18:21:49.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.717+0000 7ff1937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:49.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.717+0000 7ff1937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:49.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.976+0000 7ff1a1043640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:49.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.976+0000 7ff1a1043640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:49.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.976+0000 7ff1a1043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:21:49.977 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:49.976+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff19c0627b0 con 0x7ff19c1430c0 2024-03-25T18:21:50.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.010+0000 7ff1937fe640 1 -- 172.21.15.42:0/3115560405 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7ff1940524d0 con 0x7ff19c1430c0 2024-03-25T18:21:50.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.010+0000 7ff1937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:50.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.010+0000 7ff1937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:50.011 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:50.011 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 10 librados: watch_flush enter 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 10 librados: watch_flush exit 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 10 client.4321.objecter successfully canceled tick 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff17c06e0b0 legacy=0x7ff17c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 10 monclient: shutdown 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 >> v1:172.21.15.120:6789/0 conn(0x7ff19c1430c0 legacy=0x7ff19c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 shutdown_connections 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 >> 172.21.15.42:0/3115560405 conn(0x7ff19c0b7ea0 msgr2=0x7ff19c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:50.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 shutdown_connections 2024-03-25T18:21:50.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 1 -- 172.21.15.42:0/3115560405 wait complete. 2024-03-25T18:21:50.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:50.012+0000 7ff1a1043640 1 librados: shutdown 2024-03-25T18:21:50.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:56.832 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.946+0000 7f6a0a696640 10 monclient: get_monmap_and_config 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.946+0000 7f6a0a696640 10 monclient: build_initial_monmap 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.946+0000 7f6a0a696640 10 monclient: monmap: 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:21:56.947147+0000 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:21:56.947147+0000 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:56.947 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:21:56.948 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:21:56.948 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 1 Processor -- start 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 1 -- start start 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 10 monclient: init 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 10 monclient(hunting): picked mon.noname-c con 0x7f6a040a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.947+0000 7f6a0a696640 10 monclient(hunting): picked mon.noname-b con 0x7f6a041430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.948+0000 7f6a0a696640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a04142520 con 0x7f6a0413f710 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.948+0000 7f6a0a696640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040b28a0 con 0x7f6a040a98f0 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.948+0000 7f6a0a696640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040b2a30 con 0x7f6a041430c0 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.948+0000 7f6a0a696640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:56.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.948+0000 7f6a0a696640 10 monclient(hunting): authenticate will time out at 7432.174601s 2024-03-25T18:21:56.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.962+0000 7f6a037fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6a0413f710 0x7f6a041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33762/0 (socket says 172.21.15.42:33762) 2024-03-25T18:21:56.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.962+0000 7f6a037fe640 1 -- 172.21.15.42:0/3759590579 learned_addr learned my addr 172.21.15.42:0/3759590579 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:56.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.983+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211472394 0 0) 0x7f6a040b28a0 con 0x7f6a040a98f0 2024-03-25T18:21:56.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.983+0000 7f6a02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:56.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.983+0000 7f6a02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:56.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.983+0000 7f6a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:56.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.983+0000 7f6a02ffd640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:21:56.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.983+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec003520 con 0x7f6a040a98f0 2024-03-25T18:21:56.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.990+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554780289 0 0) 0x7f6a04142520 con 0x7f6a0413f710 2024-03-25T18:21:56.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.990+0000 7f6a02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:56.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.990+0000 7f6a02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:56.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.990+0000 7f6a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:56.992 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.990+0000 7f6a02ffd640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:21:56.992 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.990+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a040b28a0 con 0x7f6a0413f710 2024-03-25T18:21:57.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.999+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579789985 0 0) 0x7f6a040b2a30 con 0x7f6a041430c0 2024-03-25T18:21:57.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.999+0000 7f6a02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.999+0000 7f6a02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.999+0000 7f6a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.999+0000 7f6a02ffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:21:57.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:56.999+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a04142520 con 0x7f6a041430c0 2024-03-25T18:21:57.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635418629 0 0) 0x7f69ec003520 con 0x7f6a040a98f0 2024-03-25T18:21:57.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 >> v1:172.21.15.42:6790/0 conn(0x7f6a041430c0 legacy=0x7f6a0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 >> v1:172.21.15.42:6789/0 conn(0x7f6a0413f710 legacy=0x7f6a041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:57.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 1 monclient: found mon.noname-c 2024-03-25T18:21:57.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a040b44c0 con 0x7f6a040a98f0 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f69f4001690 con 0x7f6a040a98f0 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a0a696640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a0a696640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a02ffd640 10 monclient: dump: 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.004 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.005 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.005 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:57.005 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:57.005 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:57.005 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:57.005 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:57.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.002+0000 7f6a0a696640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:57.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69f40017f0 con 0x7f6a040a98f0 2024-03-25T18:21:57.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:57.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a02ffd640 1 -- 172.21.15.42:0/3759590579 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f69f4001c00 con 0x7f6a040a98f0 2024-03-25T18:21:57.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a0a696640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:57.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a0a696640 4 set_mon_vals no callback set 2024-03-25T18:21:57.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.033+0000 7f6a02ffd640 10 monclient: dump: 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:57.035 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.034+0000 7f6a0a696640 10 monclient: shutdown 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.034+0000 7f6a0a696640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.034+0000 7f6a0a696640 1 -- 172.21.15.42:0/3759590579 >> v1:172.21.15.120:6789/0 conn(0x7f6a040a98f0 legacy=0x7f6a040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.034+0000 7f6a0a696640 1 -- 172.21.15.42:0/3759590579 shutdown_connections 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.034+0000 7f6a0a696640 1 -- 172.21.15.42:0/3759590579 >> 172.21.15.42:0/3759590579 conn(0x7f6a040b7ea0 msgr2=0x7f6a040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.034+0000 7f6a0a696640 1 -- 172.21.15.42:0/3759590579 shutdown_connections 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.034+0000 7f6a0a696640 1 -- 172.21.15.42:0/3759590579 wait complete. 2024-03-25T18:21:57.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.035+0000 7f6a0a696640 10 monclient: build_initial_monmap 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.035+0000 7f6a0a696640 10 monclient: monmap: 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.035+0000 7f6a0a696640 1 librados: starting msgr at 2024-03-25T18:21:57.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.035+0000 7f6a0a696640 1 librados: starting objecter 2024-03-25T18:21:57.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.035+0000 7f6a0a696640 1 Processor -- start 2024-03-25T18:21:57.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 1 -- start start 2024-03-25T18:21:57.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 1 librados: setting wanted keys 2024-03-25T18:21:57.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 1 librados: calling monclient init 2024-03-25T18:21:57.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 10 monclient: init 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 10 monclient(hunting): picked mon.noname-b con 0x7f6a040a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 10 monclient(hunting): picked mon.noname-c con 0x7f6a041430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a04138a50 con 0x7f6a0413f710 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.036+0000 7f6a0a696640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040ab9e0 con 0x7f6a040a98f0 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.037+0000 7f6a0a696640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a041d7880 con 0x7f6a041430c0 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.037+0000 7f6a0a696640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:57.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.037+0000 7f6a0a696640 10 monclient(hunting): authenticate will time out at 7432.263641s 2024-03-25T18:21:57.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.046+0000 7f6a037fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6a0413f710 0x7f6a041d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33838/0 (socket says 172.21.15.42:33838) 2024-03-25T18:21:57.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.046+0000 7f6a037fe640 1 -- 172.21.15.42:0/3220990443 learned_addr learned my addr 172.21.15.42:0/3220990443 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:21:57.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.049+0000 7f6a00ff9640 10 client.?.objecter ms_handle_connect 0x7f6a0413f710 2024-03-25T18:21:57.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.049+0000 7f6a00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:57.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.050+0000 7f6a00ff9640 10 client.?.objecter ms_handle_connect 0x7f6a040a98f0 2024-03-25T18:21:57.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.050+0000 7f6a00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:57.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.057+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777731263 0 0) 0x7f6a04138a50 con 0x7f6a0413f710 2024-03-25T18:21:57.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.057+0000 7f6a00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.057+0000 7f6a00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.057+0000 7f6a00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.057+0000 7f6a00ff9640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:21:57.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.057+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f0004eb0 con 0x7f6a0413f710 2024-03-25T18:21:57.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.064+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998910257 0 0) 0x7f6a040ab9e0 con 0x7f6a040a98f0 2024-03-25T18:21:57.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.064+0000 7f6a00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.064+0000 7f6a00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.064+0000 7f6a00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.064+0000 7f6a00ff9640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:21:57.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.064+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a04138a50 con 0x7f6a040a98f0 2024-03-25T18:21:57.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.068+0000 7f6a00ff9640 10 client.?.objecter ms_handle_connect 0x7f6a041430c0 2024-03-25T18:21:57.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.068+0000 7f6a00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:57.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.074+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2739587604 0 0) 0x7f69f0004eb0 con 0x7f6a0413f710 2024-03-25T18:21:57.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.074+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a040ab9e0 con 0x7f6a0413f710 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.074+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f69f80019f0 con 0x7f6a0413f710 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.074+0000 7f6a00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.074+0000 7f6a00ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.074+0000 7f6a00ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:57.076 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.074+0000 7f6a00ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.077+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167832437 0 0) 0x7f6a041d7880 con 0x7f6a041430c0 2024-03-25T18:21:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.077+0000 7f6a00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.077+0000 7f6a00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.077+0000 7f6a00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.077+0000 7f6a00ff9640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:21:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.077+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f0004eb0 con 0x7f6a041430c0 2024-03-25T18:21:57.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.087+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258332899 0 0) 0x7f69f0004eb0 con 0x7f6a041430c0 2024-03-25T18:21:57.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.087+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a041d7880 con 0x7f6a041430c0 2024-03-25T18:21:57.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.087+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f69fc001bd0 con 0x7f6a041430c0 2024-03-25T18:21:57.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.087+0000 7f6a00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.087+0000 7f6a00ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.087+0000 7f6a00ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:57.089 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:57.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.088+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848584608 0 0) 0x7f6a04138a50 con 0x7f6a040a98f0 2024-03-25T18:21:57.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.088+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69f0004eb0 con 0x7f6a040a98f0 2024-03-25T18:21:57.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.088+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f69f4001b00 con 0x7f6a040a98f0 2024-03-25T18:21:57.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.088+0000 7f6a00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.088+0000 7f6a00ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:57.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.088+0000 7f6a00ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:57.090 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:57.091 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:57.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 110638150 0 0) 0x7f6a041d7880 con 0x7f6a041430c0 2024-03-25T18:21:57.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 >> v1:172.21.15.120:6789/0 conn(0x7f6a040a98f0 legacy=0x7f6a04138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 >> v1:172.21.15.42:6789/0 conn(0x7f6a0413f710 legacy=0x7f6a041d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:21:57.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 1 monclient: found mon.c 2024-03-25T18:21:57.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a041d84b0 con 0x7f6a041430c0 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 10 monclient: _finish_auth 0 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 5 monclient: authenticate success, global_id 4325 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 10 monclient: _renew_subs 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a041d51f0 con 0x7f6a041430c0 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 10 monclient: _renew_subs 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a04138e60 con 0x7f6a041430c0 2024-03-25T18:21:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.103+0000 7f6a0a696640 1 librados: init done 2024-03-25T18:21:57.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.104+0000 7f6a0a696640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:57.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.104+0000 7f6a0a696640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:57.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.104+0000 7f6a0a696640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:57.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.104+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a041d76e0 con 0x7f6a041430c0 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.431+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69fc001fe0 con 0x7f6a041430c0 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.431+0000 7f6a00ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.432+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f69fc003990 con 0x7f6a041430c0 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.432+0000 7f6a00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.432+0000 7f6a00ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.432+0000 7f6a00ff9640 10 monclient: dump: 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.433 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:21:57.434 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.434 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:21:57.434 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:21:57.434 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:21:57.434 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:21:57.434 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:21:57.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.432+0000 7f6a027fc640 4 set_mon_vals no callback set 2024-03-25T18:21:57.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.433+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f69fc019960 con 0x7f6a041430c0 2024-03-25T18:21:57.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.433+0000 7f6a03fff640 10 monclient: get_auth_request con 0x7f69f006e0c0 auth_method 0 2024-03-25T18:21:57.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.433+0000 7f6a00ff9640 10 client.4325.objecter ms_handle_connect 0x7f69f006e0c0 2024-03-25T18:21:57.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f69fc0535d0 con 0x7f6a041430c0 2024-03-25T18:21:57.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 10 client.4325.objecter ms_dispatch 0x7f6a04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:57.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:57.441 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:57.441 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-25T18:21:57.441 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69fc04f0e0 con 0x7f6a041430c0 2024-03-25T18:21:57.441 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:57.441 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.439+0000 7f6a00ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:57.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.689+0000 7f6a0a696640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:57.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.689+0000 7f6a0a696640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:57.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.689+0000 7f6a0a696640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:21:57.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.689+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a041d7a70 con 0x7f6a041430c0 2024-03-25T18:21:57.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.728+0000 7f6a00ff9640 1 -- 172.21.15.42:0/3220990443 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f69fc051010 con 0x7f6a041430c0 2024-03-25T18:21:57.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.729+0000 7f6a00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:57.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.729+0000 7f6a00ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:57.730 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:21:57.730 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 10 librados: watch_flush enter 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 10 librados: watch_flush exit 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 10 client.4325.objecter successfully canceled tick 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 >> v1:172.21.15.42:6800/3081750888 conn(0x7f69f006e0c0 legacy=0x7f69f0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 10 monclient: shutdown 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 >> v1:172.21.15.42:6790/0 conn(0x7f6a041430c0 legacy=0x7f6a041e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 shutdown_connections 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 >> 172.21.15.42:0/3220990443 conn(0x7f6a040b7ea0 msgr2=0x7f6a04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.731+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 shutdown_connections 2024-03-25T18:21:57.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.732+0000 7f6a0a696640 1 -- 172.21.15.42:0/3220990443 wait complete. 2024-03-25T18:21:57.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:21:57.732+0000 7f6a0a696640 1 librados: shutdown 2024-03-25T18:21:57.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:01.499 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:02:44.668 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.556 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:02:44.726 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.591 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:02:44.761 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.611 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:02:44.732 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.622 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:02:44.792 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.623 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:02:44.744 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.638 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:02:44.808 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.644 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:02:44.765 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:01.669 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:02:44.839 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:04.552 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.668+0000 7f3d2876d640 10 monclient: get_monmap_and_config 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.668+0000 7f3d2876d640 10 monclient: build_initial_monmap 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.668+0000 7f3d2876d640 10 monclient: monmap: 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:04.669280+0000 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:04.669280+0000 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:04.669 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:04.670 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:04.670 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:04.670 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:04.670 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:04.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.668+0000 7f3d2876d640 1 Processor -- start 2024-03-25T18:22:04.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.668+0000 7f3d2876d640 1 -- start start 2024-03-25T18:22:04.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.668+0000 7f3d2876d640 10 monclient: init 2024-03-25T18:22:04.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:04.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:04.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d200b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:04.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d200b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:04.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2014be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:04.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d200af940 con 0x7f3d200b4d60 2024-03-25T18:22:04.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d200b0a70 con 0x7f3d2014be90 2024-03-25T18:22:04.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d200b0d50 con 0x7f3d200b1730 2024-03-25T18:22:04.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:04.672 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.669+0000 7f3d2876d640 10 monclient(hunting): authenticate will time out at 7439.896266s 2024-03-25T18:22:04.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.676+0000 7f3d26d0a640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3d2014be90 0x7f3d2014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:53746/0 (socket says 172.21.15.42:53746) 2024-03-25T18:22:04.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.676+0000 7f3d26d0a640 1 -- 172.21.15.42:0/3037406047 learned_addr learned my addr 172.21.15.42:0/3037406047 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:04.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.687+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153800639 0 0) 0x7f3d200b0a70 con 0x7f3d2014be90 2024-03-25T18:22:04.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.687+0000 7f3d25507640 10 monclient(hunting): _negotiate 2024-03-25T18:22:04.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.687+0000 7f3d25507640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:04.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.687+0000 7f3d25507640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:04.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.687+0000 7f3d25507640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:22:04.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.687+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d0c003520 con 0x7f3d2014be90 2024-03-25T18:22:04.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.690+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252977238 0 0) 0x7f3d200af940 con 0x7f3d200b4d60 2024-03-25T18:22:04.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.690+0000 7f3d25507640 10 monclient(hunting): _negotiate 2024-03-25T18:22:04.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.690+0000 7f3d25507640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:04.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.690+0000 7f3d25507640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:04.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.690+0000 7f3d25507640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:22:04.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.690+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d200b0a70 con 0x7f3d200b4d60 2024-03-25T18:22:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547427001 0 0) 0x7f3d0c003520 con 0x7f3d2014be90 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 >> v1:172.21.15.42:6790/0 conn(0x7f3d200b1730 legacy=0x7f3d200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 >> v1:172.21.15.42:6789/0 conn(0x7f3d200b4d60 legacy=0x7f3d2013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 10 monclient: _finish_hunting 0 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 1 monclient: found mon.noname-c 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d200b1030 con 0x7f3d2014be90 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 10 monclient: _finish_auth 0 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 10 monclient: _check_auth_tickets 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3d18001a10 con 0x7f3d2014be90 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d2876d640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:22:04.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d2876d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d25507640 10 monclient: dump: 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:04.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.725+0000 7f3d2876d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d18001c30 con 0x7f3d2014be90 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d25507640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d25507640 1 -- 172.21.15.42:0/3037406047 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3d18001fc0 con 0x7f3d2014be90 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 4 set_mon_vals no callback set 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d25507640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d25507640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d25507640 10 monclient: dump: 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:04.745 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:04.746 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:04.746 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:04.746 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:04.746 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 10 monclient: shutdown 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 1 -- 172.21.15.42:0/3037406047 >> v1:172.21.15.120:6789/0 conn(0x7f3d2014be90 legacy=0x7f3d2014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 1 -- 172.21.15.42:0/3037406047 shutdown_connections 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 1 -- 172.21.15.42:0/3037406047 >> 172.21.15.42:0/3037406047 conn(0x7f3d200617a0 msgr2=0x7f3d20061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 1 -- 172.21.15.42:0/3037406047 shutdown_connections 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.744+0000 7f3d2876d640 1 -- 172.21.15.42:0/3037406047 wait complete. 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.745+0000 7f3d2876d640 10 monclient: build_initial_monmap 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.745+0000 7f3d2876d640 10 monclient: monmap: 2024-03-25T18:22:04.748 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.745+0000 7f3d2876d640 1 librados: starting msgr at 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.745+0000 7f3d2876d640 1 librados: starting objecter 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.745+0000 7f3d2876d640 1 Processor -- start 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 1 -- start start 2024-03-25T18:22:04.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 1 librados: setting wanted keys 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 1 librados: calling monclient init 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient: init 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d200b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d200b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d2014be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d20135b00 con 0x7f3d2014be90 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d20136270 con 0x7f3d200b4d60 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d20137c80 con 0x7f3d200b1730 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:04.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.746+0000 7f3d2876d640 10 monclient(hunting): authenticate will time out at 7439.973339s 2024-03-25T18:22:04.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.748+0000 7f3d25d08640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3d200b4d60 0x7f3d201dab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:53812/0 (socket says 172.21.15.42:53812) 2024-03-25T18:22:04.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.748+0000 7f3d25d08640 1 -- 172.21.15.42:0/958845885 learned_addr learned my addr 172.21.15.42:0/958845885 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:04.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.749+0000 7f3d16ffd640 10 client.?.objecter ms_handle_connect 0x7f3d200b4d60 2024-03-25T18:22:04.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.749+0000 7f3d16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:04.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.757+0000 7f3d16ffd640 10 client.?.objecter ms_handle_connect 0x7f3d200b1730 2024-03-25T18:22:04.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.757+0000 7f3d16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:04.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.760+0000 7f3d16ffd640 10 client.?.objecter ms_handle_connect 0x7f3d2014be90 2024-03-25T18:22:04.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.760+0000 7f3d16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:04.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.777+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277772544 0 0) 0x7f3d20135b00 con 0x7f3d2014be90 2024-03-25T18:22:04.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.777+0000 7f3d16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:04.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.777+0000 7f3d16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:04.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.777+0000 7f3d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:04.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.777+0000 7f3d16ffd640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:22:04.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.777+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d04004eb0 con 0x7f3d2014be90 2024-03-25T18:22:04.836 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.836+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674974345 0 0) 0x7f3d20136270 con 0x7f3d200b4d60 2024-03-25T18:22:04.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.836+0000 7f3d16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:04.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.836+0000 7f3d16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:04.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.836+0000 7f3d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:04.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.836+0000 7f3d16ffd640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:22:04.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.836+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20135b00 con 0x7f3d200b4d60 2024-03-25T18:22:04.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106593531 0 0) 0x7f3d20135b00 con 0x7f3d200b4d60 2024-03-25T18:22:04.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d20136270 con 0x7f3d200b4d60 2024-03-25T18:22:04.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3d100019b0 con 0x7f3d200b4d60 2024-03-25T18:22:04.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:04.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 295501019 0 0) 0x7f3d04004eb0 con 0x7f3d2014be90 2024-03-25T18:22:04.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d20135b00 con 0x7f3d2014be90 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3d18001da0 con 0x7f3d2014be90 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.844+0000 7f3d16ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:04.866 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1757025939 0 0) 0x7f3d20135b00 con 0x7f3d2014be90 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 >> v1:172.21.15.42:6790/0 conn(0x7f3d200b1730 legacy=0x7f3d2012fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 >> v1:172.21.15.120:6789/0 conn(0x7f3d200b4d60 legacy=0x7f3d201dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 1 monclient: found mon.a 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20135d50 con 0x7f3d2014be90 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 10 monclient: _renew_subs 2024-03-25T18:22:04.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d201369a0 con 0x7f3d2014be90 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 10 monclient: _renew_subs 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d20130750 con 0x7f3d2014be90 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.860+0000 7f3d2876d640 1 librados: init done 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.861+0000 7f3d2876d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.861+0000 7f3d2876d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.861+0000 7f3d2876d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:22:04.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:04.861+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d201401f0 con 0x7f3d2014be90 2024-03-25T18:22:05.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.194+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d18004b60 con 0x7f3d2014be90 2024-03-25T18:22:05.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.194+0000 7f3d16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:05.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.194+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3d18004e70 con 0x7f3d2014be90 2024-03-25T18:22:05.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.194+0000 7f3d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:05.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.194+0000 7f3d16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:05.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.194+0000 7f3d16ffd640 10 monclient: dump: 2024-03-25T18:22:05.195 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:05.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.194+0000 7f3d24d06640 4 set_mon_vals no callback set 2024-03-25T18:22:05.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.195+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f3d1801ae40 con 0x7f3d2014be90 2024-03-25T18:22:05.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.196+0000 7f3d26509640 10 monclient: get_auth_request con 0x7f3d0406e030 auth_method 0 2024-03-25T18:22:05.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.196+0000 7f3d16ffd640 10 client.4332.objecter ms_handle_connect 0x7f3d0406e030 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f3d18054960 con 0x7f3d2014be90 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 10 client.4332.objecter ms_dispatch 0x7f3d20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d180502a0 con 0x7f3d2014be90 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:05.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.202+0000 7f3d16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:05.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.462+0000 7f3d2876d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:05.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.462+0000 7f3d2876d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:05.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.462+0000 7f3d2876d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:22:05.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.462+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d200627b0 con 0x7f3d2014be90 2024-03-25T18:22:05.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.499+0000 7f3d16ffd640 1 -- 172.21.15.42:0/958845885 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f3d180523a0 con 0x7f3d2014be90 2024-03-25T18:22:05.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.499+0000 7f3d16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:05.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.499+0000 7f3d16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:05.500 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:05.501 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:05.503 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 10 librados: watch_flush enter 2024-03-25T18:22:05.503 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 10 librados: watch_flush exit 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 10 client.4332.objecter successfully canceled tick 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 >> v1:172.21.15.42:6800/3081750888 conn(0x7f3d0406e030 legacy=0x7f3d040704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 10 monclient: shutdown 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 >> v1:172.21.15.42:6789/0 conn(0x7f3d2014be90 legacy=0x7f3d201e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 shutdown_connections 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 >> 172.21.15.42:0/958845885 conn(0x7f3d200617a0 msgr2=0x7f3d2013c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 shutdown_connections 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 1 -- 172.21.15.42:0/958845885 wait complete. 2024-03-25T18:22:05.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:05.502+0000 7f3d2876d640 1 librados: shutdown 2024-03-25T18:22:05.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:12.321 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:12.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.428+0000 7fe583ee2640 10 monclient: get_monmap_and_config 2024-03-25T18:22:12.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.428+0000 7fe583ee2640 10 monclient: build_initial_monmap 2024-03-25T18:22:12.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.428+0000 7fe583ee2640 10 monclient: monmap: 2024-03-25T18:22:12.429 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:12.429823+0000 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:12.429823+0000 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:12.430 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:12.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.429+0000 7fe583ee2640 1 Processor -- start 2024-03-25T18:22:12.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.429+0000 7fe583ee2640 1 -- start start 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.429+0000 7fe583ee2640 10 monclient: init 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 10 monclient(hunting): picked mon.noname-c con 0x7fe57c0a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 10 monclient(hunting): picked mon.noname-a con 0x7fe57c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 10 monclient(hunting): picked mon.noname-b con 0x7fe57c1430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c142520 con 0x7fe57c13f710 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c0b28a0 con 0x7fe57c0a98f0 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c0b2a30 con 0x7fe57c1430c0 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.430+0000 7fe583ee2640 10 monclient(hunting): authenticate will time out at 7447.656824s 2024-03-25T18:22:12.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.432+0000 7fe581c7e640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe57c0a98f0 0x7fe57c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:51520/0 (socket says 172.21.15.42:51520) 2024-03-25T18:22:12.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.432+0000 7fe581c7e640 1 -- 172.21.15.42:0/1920083863 learned_addr learned my addr 172.21.15.42:0/1920083863 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:12.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.446+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370806157 0 0) 0x7fe57c0b28a0 con 0x7fe57c0a98f0 2024-03-25T18:22:12.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.446+0000 7fe580c7c640 10 monclient(hunting): _negotiate 2024-03-25T18:22:12.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.446+0000 7fe580c7c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:12.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.446+0000 7fe580c7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:12.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.446+0000 7fe580c7c640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:22:12.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.446+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe568003520 con 0x7fe57c0a98f0 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.460+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 816592866 0 0) 0x7fe568003520 con 0x7fe57c0a98f0 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 >> v1:172.21.15.42:6790/0 conn(0x7fe57c1430c0 legacy=0x7fe57c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 >> v1:172.21.15.42:6789/0 conn(0x7fe57c13f710 legacy=0x7fe57c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 10 monclient: _finish_hunting 0 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 1 monclient: found mon.noname-c 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c0b44c0 con 0x7fe57c0a98f0 2024-03-25T18:22:12.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 10 monclient: _finish_auth 0 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 10 monclient: _check_auth_tickets 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe578001690 con 0x7fe57c0a98f0 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe580c7c640 10 monclient: dump: 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:12.463 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:12.464 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:12.464 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:12.464 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:12.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe583ee2640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:22:12.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.461+0000 7fe583ee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5780017f0 con 0x7fe57c0a98f0 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe580c7c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe580c7c640 1 -- 172.21.15.42:0/1920083863 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe578001c40 con 0x7fe57c0a98f0 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe580c7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe580c7c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe580c7c640 10 monclient: dump: 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.487 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:12.488 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:12.488 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:12.488 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:12.488 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:12.488 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe583ee2640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:12.488 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.486+0000 7fe583ee2640 4 set_mon_vals no callback set 2024-03-25T18:22:12.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.487+0000 7fe583ee2640 10 monclient: shutdown 2024-03-25T18:22:12.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.487+0000 7fe583ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.487+0000 7fe583ee2640 1 -- 172.21.15.42:0/1920083863 >> v1:172.21.15.120:6789/0 conn(0x7fe57c0a98f0 legacy=0x7fe57c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.487+0000 7fe583ee2640 1 -- 172.21.15.42:0/1920083863 shutdown_connections 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.487+0000 7fe583ee2640 1 -- 172.21.15.42:0/1920083863 >> 172.21.15.42:0/1920083863 conn(0x7fe57c0b7ea0 msgr2=0x7fe57c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.487+0000 7fe583ee2640 1 -- 172.21.15.42:0/1920083863 shutdown_connections 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.487+0000 7fe583ee2640 1 -- 172.21.15.42:0/1920083863 wait complete. 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.488+0000 7fe583ee2640 10 monclient: build_initial_monmap 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.488+0000 7fe583ee2640 10 monclient: monmap: 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.488+0000 7fe583ee2640 1 librados: starting msgr at 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.488+0000 7fe583ee2640 1 librados: starting objecter 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.488+0000 7fe583ee2640 1 Processor -- start 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.488+0000 7fe583ee2640 1 -- start start 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.488+0000 7fe583ee2640 1 librados: setting wanted keys 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 1 librados: calling monclient init 2024-03-25T18:22:12.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient: init 2024-03-25T18:22:12.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:12.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:12.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient(hunting): picked mon.noname-b con 0x7fe57c0a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:12.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient(hunting): picked mon.noname-c con 0x7fe57c13f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:12.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient(hunting): picked mon.noname-a con 0x7fe57c1430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:12.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c1db7d0 con 0x7fe57c1430c0 2024-03-25T18:22:12.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c0b1420 con 0x7fe57c0a98f0 2024-03-25T18:22:12.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c0b15b0 con 0x7fe57c13f710 2024-03-25T18:22:12.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:12.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.489+0000 7fe583ee2640 10 monclient(hunting): authenticate will time out at 7447.716370s 2024-03-25T18:22:12.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.497+0000 7fe58147d640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fe57c13f710 0x7fe57c0b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:51132/0 (socket says 172.21.15.42:51132) 2024-03-25T18:22:12.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.497+0000 7fe58147d640 1 -- 172.21.15.42:0/200988535 learned_addr learned my addr 172.21.15.42:0/200988535 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:12.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.498+0000 7fe5727fc640 10 client.?.objecter ms_handle_connect 0x7fe57c13f710 2024-03-25T18:22:12.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.498+0000 7fe5727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:12.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.506+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557417872 0 0) 0x7fe57c0b15b0 con 0x7fe57c13f710 2024-03-25T18:22:12.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.506+0000 7fe5727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:12.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.506+0000 7fe5727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:12.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.506+0000 7fe5727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:12.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.506+0000 7fe5727fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:22:12.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.506+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe560004eb0 con 0x7fe57c13f710 2024-03-25T18:22:12.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.509+0000 7fe5727fc640 10 client.?.objecter ms_handle_connect 0x7fe57c0a98f0 2024-03-25T18:22:12.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.509+0000 7fe5727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:12.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.516+0000 7fe5727fc640 10 client.?.objecter ms_handle_connect 0x7fe57c1430c0 2024-03-25T18:22:12.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.516+0000 7fe5727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:12.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.519+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3757287 0 0) 0x7fe560004eb0 con 0x7fe57c13f710 2024-03-25T18:22:12.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.519+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe57c0b15b0 con 0x7fe57c13f710 2024-03-25T18:22:12.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.519+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe56c0019d0 con 0x7fe57c13f710 2024-03-25T18:22:12.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.519+0000 7fe5727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.519+0000 7fe5727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.519+0000 7fe5727fc640 10 monclient(hunting): dump: 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:12.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.519+0000 7fe5727fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.525+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685041239 0 0) 0x7fe57c0b1420 con 0x7fe57c0a98f0 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.525+0000 7fe5727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.525+0000 7fe5727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:12.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.525+0000 7fe5727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:12.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.525+0000 7fe5727fc640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:22:12.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.525+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe560004eb0 con 0x7fe57c0a98f0 2024-03-25T18:22:12.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4110744519 0 0) 0x7fe57c0b15b0 con 0x7fe57c13f710 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 >> v1:172.21.15.120:6789/0 conn(0x7fe57c0a98f0 legacy=0x7fe57c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 >> v1:172.21.15.42:6789/0 conn(0x7fe57c1430c0 legacy=0x7fe57c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 1 monclient: found mon.c 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c0b1740 con 0x7fe57c13f710 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe5727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:22:12.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 10 monclient: _renew_subs 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c1db1a0 con 0x7fe57c13f710 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 10 monclient: _renew_subs 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c0b08f0 con 0x7fe57c13f710 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.531+0000 7fe583ee2640 1 librados: init done 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.533+0000 7fe583ee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.533+0000 7fe583ee2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.533+0000 7fe583ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.533+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c0627b0 con 0x7fe57c13f710 2024-03-25T18:22:12.856 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.855+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe56c001c50 con 0x7fe57c13f710 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.855+0000 7fe5727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.855+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe56c003a60 con 0x7fe57c13f710 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.855+0000 7fe5727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.855+0000 7fe5727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.855+0000 7fe5727fc640 10 monclient: dump: 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:12.857 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:12.858 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:12.858 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.855+0000 7fe573fff640 4 set_mon_vals no callback set 2024-03-25T18:22:12.858 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.856+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fe56c003c80 con 0x7fe57c13f710 2024-03-25T18:22:12.858 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.857+0000 7fe581c7e640 10 monclient: get_auth_request con 0x7fe56006dd90 auth_method 0 2024-03-25T18:22:12.858 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.857+0000 7fe5727fc640 10 client.4337.objecter ms_handle_connect 0x7fe56006dd90 2024-03-25T18:22:12.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fe56c054780 con 0x7fe57c13f710 2024-03-25T18:22:12.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 10 client.4337.objecter ms_dispatch 0x7fe57c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:12.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:12.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:12.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T18:22:12.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe56c050290 con 0x7fe57c13f710 2024-03-25T18:22:12.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:12.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:12.865+0000 7fe5727fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:13.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.123+0000 7fe583ee2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:13.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.123+0000 7fe583ee2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:13.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.123+0000 7fe583ee2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:13.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.123+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe57c0abb10 con 0x7fe57c13f710 2024-03-25T18:22:13.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.148+0000 7fe5727fc640 1 -- 172.21.15.42:0/200988535 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fe56c0521c0 con 0x7fe57c13f710 2024-03-25T18:22:13.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.148+0000 7fe5727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:13.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.148+0000 7fe5727fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:13.150 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:13.150 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:13.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.150+0000 7fe583ee2640 10 librados: watch_flush enter 2024-03-25T18:22:13.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 10 librados: watch_flush exit 2024-03-25T18:22:13.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T18:22:13.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 10 client.4337.objecter successfully canceled tick 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 >> v1:172.21.15.42:6800/3081750888 conn(0x7fe56006dd90 legacy=0x7fe560070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 10 monclient: shutdown 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 >> v1:172.21.15.42:6790/0 conn(0x7fe57c13f710 legacy=0x7fe57c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 shutdown_connections 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 >> 172.21.15.42:0/200988535 conn(0x7fe57c0b7ea0 msgr2=0x7fe57c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 shutdown_connections 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 1 -- 172.21.15.42:0/200988535 wait complete. 2024-03-25T18:22:13.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:13.151+0000 7fe583ee2640 1 librados: shutdown 2024-03-25T18:22:13.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:19.970 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:20.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.084+0000 7ff063825640 10 monclient: get_monmap_and_config 2024-03-25T18:22:20.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.084+0000 7ff063825640 10 monclient: build_initial_monmap 2024-03-25T18:22:20.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.084+0000 7ff063825640 10 monclient: monmap: 2024-03-25T18:22:20.085 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:20.085 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:20.085 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:20.085558+0000 2024-03-25T18:22:20.086 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:20.085558+0000 2024-03-25T18:22:20.086 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:20.086 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:20.086 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:20.086 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:20.086 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:20.086 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.085+0000 7ff063825640 1 Processor -- start 2024-03-25T18:22:20.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.085+0000 7ff063825640 1 -- start start 2024-03-25T18:22:20.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.085+0000 7ff063825640 10 monclient: init 2024-03-25T18:22:20.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.085+0000 7ff063825640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:20.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.085+0000 7ff063825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:20.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 10 monclient(hunting): picked mon.noname-b con 0x7ff05c0b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:20.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 10 monclient(hunting): picked mon.noname-c con 0x7ff05c0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:20.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 10 monclient(hunting): picked mon.noname-a con 0x7ff05c14be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:20.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0af940 con 0x7ff05c14be90 2024-03-25T18:22:20.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0b0a70 con 0x7ff05c0b1730 2024-03-25T18:22:20.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0b0d50 con 0x7ff05c0b4d60 2024-03-25T18:22:20.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:20.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.086+0000 7ff063825640 10 monclient(hunting): authenticate will time out at 7455.312657s 2024-03-25T18:22:20.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.096+0000 7ff061dc2640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff05c14be90 0x7ff05c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39064/0 (socket says 172.21.15.42:39064) 2024-03-25T18:22:20.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.096+0000 7ff061dc2640 1 -- 172.21.15.42:0/718433944 learned_addr learned my addr 172.21.15.42:0/718433944 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:20.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.109+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617936761 0 0) 0x7ff05c0af940 con 0x7ff05c14be90 2024-03-25T18:22:20.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.109+0000 7ff053fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:20.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.109+0000 7ff053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:20.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.109+0000 7ff053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:20.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.109+0000 7ff053fff640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:22:20.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.109+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff048003520 con 0x7ff05c14be90 2024-03-25T18:22:20.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.113+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905889876 0 0) 0x7ff05c0b0d50 con 0x7ff05c0b4d60 2024-03-25T18:22:20.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.113+0000 7ff053fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:20.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.113+0000 7ff053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:20.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.113+0000 7ff053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:20.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.113+0000 7ff053fff640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:22:20.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.113+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05c0af940 con 0x7ff05c0b4d60 2024-03-25T18:22:20.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689341455 0 0) 0x7ff048003520 con 0x7ff05c14be90 2024-03-25T18:22:20.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 >> v1:172.21.15.42:6790/0 conn(0x7ff05c0b4d60 legacy=0x7ff05c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 >> v1:172.21.15.120:6789/0 conn(0x7ff05c0b1730 legacy=0x7ff05c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:20.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 1 monclient: found mon.noname-a 2024-03-25T18:22:20.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0b1030 con 0x7ff05c14be90 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff054001a10 con 0x7ff05c14be90 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff063825640 5 monclient: authenticate success, global_id 4341 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff063825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff053fff640 10 monclient: dump: 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.121 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.122 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.122 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:20.122 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:20.122 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:20.122 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:20.122 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.119+0000 7ff063825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:20.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff054001b70 con 0x7ff05c14be90 2024-03-25T18:22:20.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff053fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:20.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff053fff640 1 -- 172.21.15.42:0/718433944 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff054001f80 con 0x7ff05c14be90 2024-03-25T18:22:20.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff053fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:20.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff053fff640 10 monclient: dump: 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff063825640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:20.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.145+0000 7ff063825640 4 set_mon_vals no callback set 2024-03-25T18:22:20.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.146+0000 7ff063825640 10 monclient: shutdown 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.146+0000 7ff063825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.146+0000 7ff063825640 1 -- 172.21.15.42:0/718433944 >> v1:172.21.15.42:6789/0 conn(0x7ff05c14be90 legacy=0x7ff05c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.146+0000 7ff063825640 1 -- 172.21.15.42:0/718433944 shutdown_connections 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.146+0000 7ff063825640 1 -- 172.21.15.42:0/718433944 >> 172.21.15.42:0/718433944 conn(0x7ff05c0617a0 msgr2=0x7ff05c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.146+0000 7ff063825640 1 -- 172.21.15.42:0/718433944 shutdown_connections 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.146+0000 7ff063825640 1 -- 172.21.15.42:0/718433944 wait complete. 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 10 monclient: build_initial_monmap 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 10 monclient: monmap: 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:20.149 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 1 librados: starting msgr at 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 1 librados: starting objecter 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 1 Processor -- start 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 1 -- start start 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 1 librados: setting wanted keys 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 1 librados: calling monclient init 2024-03-25T18:22:20.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.147+0000 7ff063825640 10 monclient: init 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 10 monclient(hunting): picked mon.noname-b con 0x7ff05c0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 10 monclient(hunting): picked mon.noname-c con 0x7ff05c0b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 10 monclient(hunting): picked mon.noname-a con 0x7ff05c14be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c13e9e0 con 0x7ff05c14be90 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c13eca0 con 0x7ff05c0b1730 2024-03-25T18:22:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c1456a0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.148+0000 7ff063825640 10 monclient(hunting): authenticate will time out at 7455.375346s 2024-03-25T18:22:20.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.163+0000 7ff060dc0640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ff05c0b4d60 0x7ff05c1d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:56860/0 (socket says 172.21.15.42:56860) 2024-03-25T18:22:20.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.163+0000 7ff060dc0640 1 -- 172.21.15.42:0/4196058180 learned_addr learned my addr 172.21.15.42:0/4196058180 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:20.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.165+0000 7ff051ffb640 10 client.?.objecter ms_handle_connect 0x7ff05c0b4d60 2024-03-25T18:22:20.166 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.165+0000 7ff051ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:20.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.172+0000 7ff051ffb640 10 client.?.objecter ms_handle_connect 0x7ff05c0b1730 2024-03-25T18:22:20.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.172+0000 7ff051ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:20.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.174+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714205225 0 0) 0x7ff05c1456a0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.174+0000 7ff051ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:20.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.174+0000 7ff051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:20.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.174+0000 7ff051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:20.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.174+0000 7ff051ffb640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:22:20.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.174+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff040004eb0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.182+0000 7ff051ffb640 10 client.?.objecter ms_handle_connect 0x7ff05c14be90 2024-03-25T18:22:20.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.182+0000 7ff051ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.186+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3801305126 0 0) 0x7ff040004eb0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.187+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff05c1456a0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.187+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff04c0019d0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.187+0000 7ff051ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.187+0000 7ff051ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.187+0000 7ff051ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:20.188 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:20.189 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:20.189 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:20.189 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.187+0000 7ff051ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:20.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.197+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745073838 0 0) 0x7ff05c13eca0 con 0x7ff05c0b1730 2024-03-25T18:22:20.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.197+0000 7ff051ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:20.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.197+0000 7ff051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:20.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.197+0000 7ff051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:20.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.197+0000 7ff051ffb640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:22:20.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.197+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff040004eb0 con 0x7ff05c0b1730 2024-03-25T18:22:20.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.202+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321590817 0 0) 0x7ff05c13e9e0 con 0x7ff05c14be90 2024-03-25T18:22:20.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.202+0000 7ff051ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:20.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.202+0000 7ff051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:20.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.202+0000 7ff051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:20.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.202+0000 7ff051ffb640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:22:20.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.202+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05c13eca0 con 0x7ff05c14be90 2024-03-25T18:22:20.215 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.214+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2234305143 0 0) 0x7ff040004eb0 con 0x7ff05c0b1730 2024-03-25T18:22:20.215 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.214+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff05c13e9e0 con 0x7ff05c0b1730 2024-03-25T18:22:20.215 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.214+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff058001930 con 0x7ff05c0b1730 2024-03-25T18:22:20.215 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.214+0000 7ff051ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.215 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.214+0000 7ff051ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:20.215 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.214+0000 7ff051ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:20.215 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.215+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126892781 0 0) 0x7ff05c13eca0 con 0x7ff05c14be90 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.215+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff040004eb0 con 0x7ff05c14be90 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.215+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff054001e80 con 0x7ff05c14be90 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.215+0000 7ff051ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.215+0000 7ff051ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.215+0000 7ff051ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:20.217 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:20.218 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:20.218 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2685604753 0 0) 0x7ff05c1456a0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 >> v1:172.21.15.120:6789/0 conn(0x7ff05c0b1730 legacy=0x7ff05c13b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 >> v1:172.21.15.42:6789/0 conn(0x7ff05c14be90 legacy=0x7ff05c1e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:20.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 1 monclient: found mon.c 2024-03-25T18:22:20.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:20.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c132230 con 0x7ff05c0b4d60 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff051ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff063825640 5 monclient: authenticate success, global_id 4343 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff063825640 10 monclient: _renew_subs 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff063825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.216+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05c1324e0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.217+0000 7ff063825640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.217+0000 7ff063825640 10 monclient: _renew_subs 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.217+0000 7ff063825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.217+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05c13bc60 con 0x7ff05c0b4d60 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.217+0000 7ff063825640 1 librados: init done 2024-03-25T18:22:20.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.217+0000 7ff063825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:20.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.218+0000 7ff063825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:20.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.218+0000 7ff063825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:20.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.218+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05c0627b0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.242+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff04c001c50 con 0x7ff05c0b4d60 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.242+0000 7ff051ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.242+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff04c003a60 con 0x7ff05c0b4d60 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.242+0000 7ff051ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.242+0000 7ff051ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.242+0000 7ff051ffb640 10 monclient: dump: 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:20.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.242+0000 7ff0537fe640 4 set_mon_vals no callback set 2024-03-25T18:22:20.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.243+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff04c003c80 con 0x7ff05c0b4d60 2024-03-25T18:22:20.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.243+0000 7ff0615c1640 10 monclient: get_auth_request con 0x7ff04006e180 auth_method 0 2024-03-25T18:22:20.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.244+0000 7ff051ffb640 10 client.4343.objecter ms_handle_connect 0x7ff04006e180 2024-03-25T18:22:20.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7ff04c054860 con 0x7ff05c0b4d60 2024-03-25T18:22:20.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 10 client.4343.objecter ms_dispatch 0x7ff05c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:20.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:20.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:20.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-25T18:22:20.536 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff04c050370 con 0x7ff05c0b4d60 2024-03-25T18:22:20.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:20.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.535+0000 7ff051ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:20.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.798+0000 7ff063825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:20.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.798+0000 7ff063825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:20.800 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.798+0000 7ff063825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:20.800 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.798+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff05c069f40 con 0x7ff05c0b4d60 2024-03-25T18:22:20.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.970+0000 7ff051ffb640 1 -- 172.21.15.42:0/4196058180 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7ff04c0522a0 con 0x7ff05c0b4d60 2024-03-25T18:22:20.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.970+0000 7ff051ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:20.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.970+0000 7ff051ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:20.971 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:20.972 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:20.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.972+0000 7ff063825640 10 librados: watch_flush enter 2024-03-25T18:22:20.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.972+0000 7ff063825640 10 librados: watch_flush exit 2024-03-25T18:22:20.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.972+0000 7ff063825640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-25T18:22:20.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.972+0000 7ff063825640 10 client.4343.objecter successfully canceled tick 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff04006e180 legacy=0x7ff040070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 10 monclient: shutdown 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 >> v1:172.21.15.42:6790/0 conn(0x7ff05c0b4d60 legacy=0x7ff05c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 shutdown_connections 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 >> 172.21.15.42:0/4196058180 conn(0x7ff05c0617a0 msgr2=0x7ff05c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 shutdown_connections 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 1 -- 172.21.15.42:0/4196058180 wait complete. 2024-03-25T18:22:20.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:20.973+0000 7ff063825640 1 librados: shutdown 2024-03-25T18:22:20.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:21.618 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:04.787 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.624 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:04.794 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.641 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:04.811 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.651 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:04.821 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.670 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:04.790 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.674 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:04.794 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.694 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:04.863 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.709 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:04.829 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.717 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:04.887 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:27.792 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:27.908 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.907+0000 7f7c8d0c8640 10 monclient: get_monmap_and_config 2024-03-25T18:22:27.908 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.907+0000 7f7c8d0c8640 10 monclient: build_initial_monmap 2024-03-25T18:22:27.908 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.907+0000 7f7c8d0c8640 10 monclient: monmap: 2024-03-25T18:22:27.908 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:27.908 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:27.908 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:27.908484+0000 2024-03-25T18:22:27.908 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:27.908484+0000 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.908+0000 7f7c8d0c8640 1 Processor -- start 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.908+0000 7f7c8d0c8640 1 -- start start 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.908+0000 7f7c8d0c8640 10 monclient: init 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f7c880a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f7c8813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f7c881430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c88142520 con 0x7f7c8813f710 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880b28a0 con 0x7f7c881430c0 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c880b2a30 con 0x7f7c880a98f0 2024-03-25T18:22:27.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:27.911 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.909+0000 7f7c8d0c8640 10 monclient(hunting): authenticate will time out at 7463.135747s 2024-03-25T18:22:27.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.913+0000 7f7c8759e640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f7c881430c0 0x7f7c8814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:57780/0 (socket says 172.21.15.42:57780) 2024-03-25T18:22:27.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.913+0000 7f7c8759e640 1 -- 172.21.15.42:0/326989843 learned_addr learned my addr 172.21.15.42:0/326989843 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:27.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.920+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970927177 0 0) 0x7f7c880b28a0 con 0x7f7c881430c0 2024-03-25T18:22:27.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.920+0000 7f7c85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.920+0000 7f7c85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.920+0000 7f7c85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.920+0000 7f7c85d9b640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:22:27.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.920+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c70003520 con 0x7f7c881430c0 2024-03-25T18:22:27.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.928+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511175743 0 0) 0x7f7c88142520 con 0x7f7c8813f710 2024-03-25T18:22:27.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.928+0000 7f7c85d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.928+0000 7f7c85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.928+0000 7f7c85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.928+0000 7f7c85d9b640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:22:27.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.928+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c880b28a0 con 0x7f7c8813f710 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3732210278 0 0) 0x7f7c70003520 con 0x7f7c881430c0 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 >> v1:172.21.15.42:6790/0 conn(0x7f7c880a98f0 legacy=0x7f7c880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 >> v1:172.21.15.42:6789/0 conn(0x7f7c8813f710 legacy=0x7f7c881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 1 monclient: found mon.noname-c 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880b44c0 con 0x7f7c881430c0 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 10 monclient: _finish_auth 0 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7c80001a10 con 0x7f7c881430c0 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c85d9b640 10 monclient: dump: 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c8d0c8640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:22:27.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.931+0000 7f7c8d0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c80001b70 con 0x7f7c881430c0 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c85d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c85d9b640 1 -- 172.21.15.42:0/326989843 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7c80001f80 con 0x7f7c881430c0 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 4 set_mon_vals no callback set 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c85d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c85d9b640 10 monclient: dump: 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:27.939 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:27.940 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:27.940 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:27.940 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 10 monclient: shutdown 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/326989843 >> v1:172.21.15.120:6789/0 conn(0x7f7c881430c0 legacy=0x7f7c8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/326989843 shutdown_connections 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/326989843 >> 172.21.15.42:0/326989843 conn(0x7f7c880b7ea0 msgr2=0x7f7c880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.938+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/326989843 shutdown_connections 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.939+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/326989843 wait complete. 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 10 monclient: build_initial_monmap 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 10 monclient: monmap: 2024-03-25T18:22:27.941 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 1 librados: starting msgr at 2024-03-25T18:22:27.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 1 librados: starting objecter 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 1 Processor -- start 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 1 -- start start 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 1 librados: setting wanted keys 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 1 librados: calling monclient init 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.940+0000 7f7c8d0c8640 10 monclient: init 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f7c880a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f7c8813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f7c881430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c881db170 con 0x7f7c881430c0 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c881dac50 con 0x7f7c880a98f0 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c881dade0 con 0x7f7c8813f710 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:27.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.941+0000 7f7c8d0c8640 10 monclient(hunting): authenticate will time out at 7463.168398s 2024-03-25T18:22:27.947 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.946+0000 7f7c8759e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7c881430c0 0x7f7c88132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41316/0 (socket says 172.21.15.42:41316) 2024-03-25T18:22:27.947 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.946+0000 7f7c8759e640 1 -- 172.21.15.42:0/3550032611 learned_addr learned my addr 172.21.15.42:0/3550032611 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.947+0000 7f7c6f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c881430c0 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.947+0000 7f7c6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:27.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.951+0000 7f7c6f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c8813f710 2024-03-25T18:22:27.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.951+0000 7f7c6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:27.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.955+0000 7f7c6f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c880a98f0 2024-03-25T18:22:27.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.955+0000 7f7c6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:27.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.967+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911181926 0 0) 0x7f7c881dac50 con 0x7f7c880a98f0 2024-03-25T18:22:27.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.967+0000 7f7c6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.967+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.967+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.967+0000 7f7c6f7fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:22:27.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.967+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c74004eb0 con 0x7f7c880a98f0 2024-03-25T18:22:27.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.969+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863003339 0 0) 0x7f7c881db170 con 0x7f7c881430c0 2024-03-25T18:22:27.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.969+0000 7f7c6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.969+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.969+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.969+0000 7f7c6f7fe640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.969+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c881dac50 con 0x7f7c881430c0 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.970+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376923938 0 0) 0x7f7c881dade0 con 0x7f7c8813f710 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.970+0000 7f7c6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.970+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.970+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.970+0000 7f7c6f7fe640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:22:27.971 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.970+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c881db170 con 0x7f7c8813f710 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.982+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3378517110 0 0) 0x7f7c74004eb0 con 0x7f7c880a98f0 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.982+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c881dade0 con 0x7f7c880a98f0 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.982+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7c78001930 con 0x7f7c880a98f0 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.982+0000 7f7c6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.982+0000 7f7c6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.982+0000 7f7c6f7fe640 10 monclient(hunting): dump: 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.983 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:27.984 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:27.984 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:27.984 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:27.984 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:27.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.982+0000 7f7c6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.988+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1892268954 0 0) 0x7f7c881dac50 con 0x7f7c881430c0 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.988+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c74004eb0 con 0x7f7c881430c0 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.988+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7c80001e80 con 0x7f7c881430c0 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.988+0000 7f7c6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.988+0000 7f7c6f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.988+0000 7f7c6f7fe640 10 monclient(hunting): dump: 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:27.989 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:27.990 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:27.990 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:27.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2641071612 0 0) 0x7f7c881dade0 con 0x7f7c880a98f0 2024-03-25T18:22:27.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 >> v1:172.21.15.42:6790/0 conn(0x7f7c8813f710 legacy=0x7f7c881d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 >> v1:172.21.15.42:6789/0 conn(0x7f7c881430c0 legacy=0x7f7c88132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 1 monclient: found mon.b 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8813c170 con 0x7f7c880a98f0 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 5 monclient: authenticate success, global_id 4351 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 10 monclient: _renew_subs 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8813c420 con 0x7f7c880a98f0 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:27.995 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 10 monclient: _renew_subs 2024-03-25T18:22:27.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:27.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88133860 con 0x7f7c880a98f0 2024-03-25T18:22:27.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.992+0000 7f7c8d0c8640 1 librados: init done 2024-03-25T18:22:27.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.993+0000 7f7c8d0c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:27.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.993+0000 7f7c8d0c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:27.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.993+0000 7f7c8d0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:27.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:27.993+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c881db300 con 0x7f7c880a98f0 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.269+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c780035f0 con 0x7f7c880a98f0 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.269+0000 7f7c6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.269+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7c78003900 con 0x7f7c880a98f0 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.269+0000 7f7c6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.269+0000 7f7c6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.269+0000 7f7c6f7fe640 10 monclient: dump: 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:28.270 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:28.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.269+0000 7f7c8559a640 4 set_mon_vals no callback set 2024-03-25T18:22:28.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.270+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f7c780198d0 con 0x7f7c880a98f0 2024-03-25T18:22:28.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.270+0000 7f7c8659c640 10 monclient: get_auth_request con 0x7f7c7406df20 auth_method 0 2024-03-25T18:22:28.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.270+0000 7f7c6f7fe640 10 client.4351.objecter ms_handle_connect 0x7f7c7406df20 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f7c78053650 con 0x7f7c880a98f0 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 10 client.4351.objecter ms_dispatch 0x7f7c88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c7804ef90 con 0x7f7c880a98f0 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:28.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.275+0000 7f7c6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:28.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.493+0000 7f7c8d0c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:28.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.493+0000 7f7c8d0c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:28.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.493+0000 7f7c8d0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:28.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.493+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c881daf70 con 0x7f7c880a98f0 2024-03-25T18:22:28.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.513+0000 7f7c6f7fe640 1 -- 172.21.15.42:0/3550032611 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f7c78051090 con 0x7f7c880a98f0 2024-03-25T18:22:28.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.513+0000 7f7c6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:28.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.513+0000 7f7c6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:28.514 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:28.515 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 10 librados: watch_flush enter 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 10 librados: watch_flush exit 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 10 client.4351.objecter successfully canceled tick 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 >> v1:172.21.15.42:6800/3081750888 conn(0x7f7c7406df20 legacy=0x7f7c740703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 10 monclient: shutdown 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.515+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 >> v1:172.21.15.120:6789/0 conn(0x7f7c880a98f0 legacy=0x7f7c881394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.516+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 shutdown_connections 2024-03-25T18:22:28.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.516+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 >> 172.21.15.42:0/3550032611 conn(0x7f7c880b7ea0 msgr2=0x7f7c8814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:28.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.516+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 shutdown_connections 2024-03-25T18:22:28.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.516+0000 7f7c8d0c8640 1 -- 172.21.15.42:0/3550032611 wait complete. 2024-03-25T18:22:28.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:28.516+0000 7f7c8d0c8640 1 librados: shutdown 2024-03-25T18:22:28.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:35.335 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.448+0000 7f0d2a231640 10 monclient: get_monmap_and_config 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.448+0000 7f0d2a231640 10 monclient: build_initial_monmap 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.448+0000 7f0d2a231640 10 monclient: monmap: 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:35.449306+0000 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:35.449306+0000 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:35.449 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:35.450 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:35.450 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.449+0000 7f0d2a231640 1 Processor -- start 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.449+0000 7f0d2a231640 1 -- start start 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.449+0000 7f0d2a231640 10 monclient: init 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 10 monclient(hunting): picked mon.noname-c con 0x7f0d2406ab80 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 10 monclient(hunting): picked mon.noname-b con 0x7f0d2406af60 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 10 monclient(hunting): picked mon.noname-a con 0x7f0d240b00b0 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240b0500 con 0x7f0d240b00b0 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2413ffa0 con 0x7f0d2406ab80 2024-03-25T18:22:35.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240af520 con 0x7f0d2406af60 2024-03-25T18:22:35.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:35.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.450+0000 7f0d2a231640 10 monclient(hunting): authenticate will time out at 7470.676747s 2024-03-25T18:22:35.455 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.454+0000 7f0d22ffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f0d2406af60 0x7f0d2413f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:38156/0 (socket says 172.21.15.42:38156) 2024-03-25T18:22:35.455 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.454+0000 7f0d22ffd640 1 -- 172.21.15.42:0/1234219365 learned_addr learned my addr 172.21.15.42:0/1234219365 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:35.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.467+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262044807 0 0) 0x7f0d240af520 con 0x7f0d2406af60 2024-03-25T18:22:35.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.467+0000 7f0d227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.467+0000 7f0d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.467+0000 7f0d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.467+0000 7f0d227fc640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:22:35.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.467+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c003520 con 0x7f0d2406af60 2024-03-25T18:22:35.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.468+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812490245 0 0) 0x7f0d240b0500 con 0x7f0d240b00b0 2024-03-25T18:22:35.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.468+0000 7f0d227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.468+0000 7f0d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.468+0000 7f0d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.468+0000 7f0d227fc640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:22:35.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.468+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d240af520 con 0x7f0d240b00b0 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2575271389 0 0) 0x7f0d0c003520 con 0x7f0d2406af60 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 >> v1:172.21.15.120:6789/0 conn(0x7f0d2406ab80 legacy=0x7f0d240ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 >> v1:172.21.15.42:6789/0 conn(0x7f0d240b00b0 legacy=0x7f0d2414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 1 monclient: found mon.noname-b 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d24072890 con 0x7f0d2406af60 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.472+0000 7f0d227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.473+0000 7f0d2a231640 5 monclient: authenticate success, global_id 4352 2024-03-25T18:22:35.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.473+0000 7f0d2a231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:35.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.474+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d18001650 con 0x7f0d2406af60 2024-03-25T18:22:35.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.474+0000 7f0d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.474+0000 7f0d227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:35.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.474+0000 7f0d227fc640 10 monclient: dump: 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:35.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.474+0000 7f0d2a231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:35.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d180018a0 con 0x7f0d2406af60 2024-03-25T18:22:35.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d227fc640 1 -- 172.21.15.42:0/1234219365 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d18001c30 con 0x7f0d2406af60 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d227fc640 10 monclient: dump: 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:35.482 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:35.483 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:35.483 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d2a231640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:35.483 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.480+0000 7f0d2a231640 4 set_mon_vals no callback set 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.481+0000 7f0d2a231640 10 monclient: shutdown 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.481+0000 7f0d2a231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.481+0000 7f0d2a231640 1 -- 172.21.15.42:0/1234219365 >> v1:172.21.15.42:6790/0 conn(0x7f0d2406af60 legacy=0x7f0d2413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.481+0000 7f0d2a231640 1 -- 172.21.15.42:0/1234219365 shutdown_connections 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.481+0000 7f0d2a231640 1 -- 172.21.15.42:0/1234219365 >> 172.21.15.42:0/1234219365 conn(0x7f0d240b4a80 msgr2=0x7f0d240b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.481+0000 7f0d2a231640 1 -- 172.21.15.42:0/1234219365 shutdown_connections 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.481+0000 7f0d2a231640 1 -- 172.21.15.42:0/1234219365 wait complete. 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.482+0000 7f0d2a231640 10 monclient: build_initial_monmap 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.482+0000 7f0d2a231640 10 monclient: monmap: 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:35.485 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.482+0000 7f0d2a231640 1 librados: starting msgr at 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.482+0000 7f0d2a231640 1 librados: starting objecter 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.482+0000 7f0d2a231640 1 Processor -- start 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 1 -- start start 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 1 librados: setting wanted keys 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 1 librados: calling monclient init 2024-03-25T18:22:35.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient: init 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient(hunting): picked mon.noname-a con 0x7f0d2406ab80 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient(hunting): picked mon.noname-c con 0x7f0d240b00b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient(hunting): picked mon.noname-b con 0x7f0d241d73b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d24145500 con 0x7f0d2406ab80 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d241d7890 con 0x7f0d241d73b0 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d241d4f20 con 0x7f0d240b00b0 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:35.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.483+0000 7f0d2a231640 10 monclient(hunting): authenticate will time out at 7470.710490s 2024-03-25T18:22:35.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.492+0000 7f0d23fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f0d241d73b0 0x7f0d241e2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34334/0 (socket says 172.21.15.42:34334) 2024-03-25T18:22:35.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.492+0000 7f0d23fff640 1 -- 172.21.15.42:0/1323052837 learned_addr learned my addr 172.21.15.42:0/1323052837 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:35.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.493+0000 7f0d0bfff640 10 client.?.objecter ms_handle_connect 0x7f0d241d73b0 2024-03-25T18:22:35.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.493+0000 7f0d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:35.495 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.494+0000 7f0d0bfff640 10 client.?.objecter ms_handle_connect 0x7f0d2406ab80 2024-03-25T18:22:35.495 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.494+0000 7f0d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.498+0000 7f0d0bfff640 10 client.?.objecter ms_handle_connect 0x7f0d240b00b0 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.498+0000 7f0d0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:35.501 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.500+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037748342 0 0) 0x7f0d241d7890 con 0x7f0d241d73b0 2024-03-25T18:22:35.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.500+0000 7f0d0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.500+0000 7f0d0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.500+0000 7f0d0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.500+0000 7f0d0bfff640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:22:35.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.500+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d10004eb0 con 0x7f0d241d73b0 2024-03-25T18:22:35.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.508+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559480575 0 0) 0x7f0d241d4f20 con 0x7f0d240b00b0 2024-03-25T18:22:35.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.508+0000 7f0d0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.508+0000 7f0d0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.508+0000 7f0d0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.508+0000 7f0d0bfff640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:22:35.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.508+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d241d7890 con 0x7f0d240b00b0 2024-03-25T18:22:35.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.510+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671038925 0 0) 0x7f0d24145500 con 0x7f0d2406ab80 2024-03-25T18:22:35.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.510+0000 7f0d0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.510+0000 7f0d0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.510+0000 7f0d0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.510+0000 7f0d0bfff640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:22:35.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.510+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d241d4f20 con 0x7f0d2406ab80 2024-03-25T18:22:35.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.518+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2800831653 0 0) 0x7f0d10004eb0 con 0x7f0d241d73b0 2024-03-25T18:22:35.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.518+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d24145500 con 0x7f0d241d73b0 2024-03-25T18:22:35.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.518+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d1c001c90 con 0x7f0d241d73b0 2024-03-25T18:22:35.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.518+0000 7f0d0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.518+0000 7f0d0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.518+0000 7f0d0bfff640 10 monclient(hunting): dump: 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.518+0000 7f0d0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.536+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103328218 0 0) 0x7f0d241d4f20 con 0x7f0d2406ab80 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.536+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d10004eb0 con 0x7f0d2406ab80 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.536+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d14001860 con 0x7f0d2406ab80 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.536+0000 7f0d0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.536+0000 7f0d0bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.536+0000 7f0d0bfff640 10 monclient(hunting): dump: 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:35.537 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:35.538 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:35.538 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:35.538 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3316567617 0 0) 0x7f0d24145500 con 0x7f0d241d73b0 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 >> v1:172.21.15.42:6790/0 conn(0x7f0d240b00b0 legacy=0x7f0d241d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 >> v1:172.21.15.42:6789/0 conn(0x7f0d2406ab80 legacy=0x7f0d241da8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 1 monclient: found mon.b 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d241436d0 con 0x7f0d241d73b0 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 10 monclient: _finish_auth 0 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.539+0000 7f0d2a231640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 10 monclient: _renew_subs 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d241439a0 con 0x7f0d241d73b0 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 10 monclient: _renew_subs 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d24142f80 con 0x7f0d241d73b0 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.540+0000 7f0d2a231640 1 librados: init done 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.541+0000 7f0d2a231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.541+0000 7f0d2a231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.541+0000 7f0d2a231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.541+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d24143230 con 0x7f0d241d73b0 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.815+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d1c003990 con 0x7f0d241d73b0 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.815+0000 7f0d0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.815+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0d1c003ce0 con 0x7f0d241d73b0 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.815+0000 7f0d0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.815+0000 7f0d0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.815+0000 7f0d0bfff640 10 monclient: dump: 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:35.817 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:35.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.815+0000 7f0d21ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:35.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.817+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f0d1c003f00 con 0x7f0d241d73b0 2024-03-25T18:22:35.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.817+0000 7f0d237fe640 10 monclient: get_auth_request con 0x7f0d1006df70 auth_method 0 2024-03-25T18:22:35.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.817+0000 7f0d0bfff640 10 client.4357.objecter ms_handle_connect 0x7f0d1006df70 2024-03-25T18:22:35.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.822+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f0d1c054ad0 con 0x7f0d241d73b0 2024-03-25T18:22:35.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.822+0000 7f0d0bfff640 10 client.4357.objecter ms_dispatch 0x7f0d24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:35.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.822+0000 7f0d0bfff640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:35.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.822+0000 7f0d0bfff640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:35.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.822+0000 7f0d0bfff640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-25T18:22:35.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.823+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d1c0505e0 con 0x7f0d241d73b0 2024-03-25T18:22:35.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.823+0000 7f0d0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:35.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:35.823+0000 7f0d0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:36.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.050+0000 7f0d2a231640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:36.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.050+0000 7f0d2a231640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:36.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.050+0000 7f0d2a231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:36.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.050+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d241d7a20 con 0x7f0d241d73b0 2024-03-25T18:22:36.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.123+0000 7f0d0bfff640 1 -- 172.21.15.42:0/1323052837 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f0d1c052510 con 0x7f0d241d73b0 2024-03-25T18:22:36.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.123+0000 7f0d0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:36.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.123+0000 7f0d0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:36.124 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:36.124 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 10 librados: watch_flush enter 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 10 librados: watch_flush exit 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 10 client.4357.objecter successfully canceled tick 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 >> v1:172.21.15.42:6800/3081750888 conn(0x7f0d1006df70 legacy=0x7f0d10070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 10 monclient: shutdown 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 >> v1:172.21.15.120:6789/0 conn(0x7f0d241d73b0 legacy=0x7f0d241e2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 shutdown_connections 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 >> 172.21.15.42:0/1323052837 conn(0x7f0d240b4a80 msgr2=0x7f0d2413f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 shutdown_connections 2024-03-25T18:22:36.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 1 -- 172.21.15.42:0/1323052837 wait complete. 2024-03-25T18:22:36.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:36.125+0000 7f0d2a231640 1 librados: shutdown 2024-03-25T18:22:36.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:41.662 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:24.831 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.717 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:24.886 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.746 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:24.915 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.768 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:24.888 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.776 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:24.896 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.794 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:24.964 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.796 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:24.916 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.808 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:24.978 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.846 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:25.016 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:42.946 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.062+0000 7f3f7fc60640 10 monclient: get_monmap_and_config 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.062+0000 7f3f7fc60640 10 monclient: build_initial_monmap 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.062+0000 7f3f7fc60640 10 monclient: monmap: 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:43.064003+0000 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:43.064003+0000 2024-03-25T18:22:43.064 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:43.065 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:43.065 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:43.065 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:43.065 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:43.065 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.063+0000 7f3f7fc60640 1 Processor -- start 2024-03-25T18:22:43.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.063+0000 7f3f7fc60640 1 -- start start 2024-03-25T18:22:43.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.063+0000 7f3f7fc60640 10 monclient: init 2024-03-25T18:22:43.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.064+0000 7f3f7fc60640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:43.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.064+0000 7f3f7fc60640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:43.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.064+0000 7f3f7fc60640 10 monclient(hunting): picked mon.noname-a con 0x7f3f780a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:43.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.064+0000 7f3f7fc60640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:43.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.065+0000 7f3f7fc60640 10 monclient(hunting): picked mon.noname-c con 0x7f3f781430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:43.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.065+0000 7f3f7fc60640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f78142520 con 0x7f3f780a98f0 2024-03-25T18:22:43.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.065+0000 7f3f7fc60640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f780b28a0 con 0x7f3f781430c0 2024-03-25T18:22:43.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.065+0000 7f3f7fc60640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f780b2a30 con 0x7f3f7813f710 2024-03-25T18:22:43.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.065+0000 7f3f7fc60640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:43.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.065+0000 7f3f7fc60640 10 monclient(hunting): authenticate will time out at 7478.291666s 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.074+0000 7f3f7e1fd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3f781430c0 0x7f3f7814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:47048/0 (socket says 172.21.15.42:47048) 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.074+0000 7f3f7e1fd640 1 -- 172.21.15.42:0/1654771300 learned_addr learned my addr 172.21.15.42:0/1654771300 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:43.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.085+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591954441 0 0) 0x7f3f780b28a0 con 0x7f3f781430c0 2024-03-25T18:22:43.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.085+0000 7f3f7c9fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.085+0000 7f3f7c9fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.085+0000 7f3f7c9fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.085+0000 7f3f7c9fa640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:22:43.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.085+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f64003520 con 0x7f3f781430c0 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.098+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526043601 0 0) 0x7f3f64003520 con 0x7f3f781430c0 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 >> v1:172.21.15.42:6790/0 conn(0x7f3f7813f710 legacy=0x7f3f781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 >> v1:172.21.15.42:6789/0 conn(0x7f3f780a98f0 legacy=0x7f3f780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 1 monclient: found mon.noname-c 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f780b44c0 con 0x7f3f781430c0 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:43.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f70001a10 con 0x7f3f781430c0 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7fc60640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7c9fa640 10 monclient: dump: 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:43.101 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:43.102 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:43.102 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:43.102 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.102 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.099+0000 7f3f7fc60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f70001b70 con 0x7f3f781430c0 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7c9fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7c9fa640 1 -- 172.21.15.42:0/1654771300 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f70001f80 con 0x7f3f781430c0 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7fc60640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7fc60640 4 set_mon_vals no callback set 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7c9fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7c9fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.112+0000 7f3f7c9fa640 10 monclient: dump: 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:43.113 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:43.114 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.113+0000 7f3f7fc60640 10 monclient: shutdown 2024-03-25T18:22:43.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.113+0000 7f3f7fc60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:43.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.113+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1654771300 >> v1:172.21.15.120:6789/0 conn(0x7f3f781430c0 legacy=0x7f3f7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.113+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1654771300 shutdown_connections 2024-03-25T18:22:43.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.113+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1654771300 >> 172.21.15.42:0/1654771300 conn(0x7f3f780b7ea0 msgr2=0x7f3f780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.113+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1654771300 shutdown_connections 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.113+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1654771300 wait complete. 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 10 monclient: build_initial_monmap 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 10 monclient: monmap: 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:43.117 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 1 librados: starting msgr at 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 1 librados: starting objecter 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 1 Processor -- start 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 1 -- start start 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 1 librados: setting wanted keys 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 1 librados: calling monclient init 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.114+0000 7f3f7fc60640 10 monclient: init 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 10 monclient(hunting): picked mon.noname-c con 0x7f3f780a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 10 monclient(hunting): picked mon.noname-a con 0x7f3f781430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:43.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f781396e0 con 0x7f3f781430c0 2024-03-25T18:22:43.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f78138530 con 0x7f3f7813f710 2024-03-25T18:22:43.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f781399c0 con 0x7f3f780a98f0 2024-03-25T18:22:43.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:43.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.115+0000 7f3f7fc60640 10 monclient(hunting): authenticate will time out at 7478.341874s 2024-03-25T18:22:43.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.117+0000 7f3f7d1fb640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3f7813f710 0x7f3f781da850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:47084/0 (socket says 172.21.15.42:47084) 2024-03-25T18:22:43.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.117+0000 7f3f7d1fb640 1 -- 172.21.15.42:0/1636301052 learned_addr learned my addr 172.21.15.42:0/1636301052 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:43.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.118+0000 7f3f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f3f7813f710 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.118+0000 7f3f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:43.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.121+0000 7f3f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f3f781430c0 2024-03-25T18:22:43.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.121+0000 7f3f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:43.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831501506 0 0) 0x7f3f78138530 con 0x7f3f7813f710 2024-03-25T18:22:43.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:22:43.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f5c004eb0 con 0x7f3f7813f710 2024-03-25T18:22:43.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f3f780a98f0 2024-03-25T18:22:43.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.126+0000 7f3f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.130+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116501928 0 0) 0x7f3f781396e0 con 0x7f3f781430c0 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.130+0000 7f3f6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.130+0000 7f3f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.130+0000 7f3f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.130+0000 7f3f6e7fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.130+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f78138530 con 0x7f3f781430c0 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.133+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52576372 0 0) 0x7f3f781399c0 con 0x7f3f780a98f0 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.133+0000 7f3f6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.133+0000 7f3f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.133+0000 7f3f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.133+0000 7f3f6e7fc640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.133+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f781396e0 con 0x7f3f780a98f0 2024-03-25T18:22:43.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.136+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 261999216 0 0) 0x7f3f5c004eb0 con 0x7f3f7813f710 2024-03-25T18:22:43.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.136+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f781399c0 con 0x7f3f7813f710 2024-03-25T18:22:43.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.136+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f68001a30 con 0x7f3f7813f710 2024-03-25T18:22:43.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.136+0000 7f3f6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.136+0000 7f3f6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:43.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.136+0000 7f3f6e7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:43.137 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.136+0000 7f3f6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:43.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.138+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3734417409 0 0) 0x7f3f78138530 con 0x7f3f781430c0 2024-03-25T18:22:43.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.138+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f5c004eb0 con 0x7f3f781430c0 2024-03-25T18:22:43.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.138+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f70001e80 con 0x7f3f781430c0 2024-03-25T18:22:43.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.138+0000 7f3f6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.138+0000 7f3f6e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.138+0000 7f3f6e7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:43.140 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.143+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428150254 0 0) 0x7f3f781396e0 con 0x7f3f780a98f0 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.143+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f78138530 con 0x7f3f780a98f0 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.143+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f74001830 con 0x7f3f780a98f0 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.143+0000 7f3f6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.143+0000 7f3f6e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.143+0000 7f3f6e7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:43.144 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:43.145 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1233249821 0 0) 0x7f3f781399c0 con 0x7f3f7813f710 2024-03-25T18:22:43.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 >> v1:172.21.15.42:6790/0 conn(0x7f3f780a98f0 legacy=0x7f3f78137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 >> v1:172.21.15.42:6789/0 conn(0x7f3f781430c0 legacy=0x7f3f781e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 1 monclient: found mon.b 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f78138a50 con 0x7f3f7813f710 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 10 monclient: _renew_subs 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f781380a0 con 0x7f3f7813f710 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:43.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 10 monclient: _renew_subs 2024-03-25T18:22:43.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:43.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f781db140 con 0x7f3f7813f710 2024-03-25T18:22:43.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.146+0000 7f3f7fc60640 1 librados: init done 2024-03-25T18:22:43.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.147+0000 7f3f7fc60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:43.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.147+0000 7f3f7fc60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:43.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.147+0000 7f3f7fc60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:43.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.147+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f78138ca0 con 0x7f3f7813f710 2024-03-25T18:22:43.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.169+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f68001c70 con 0x7f3f7813f710 2024-03-25T18:22:43.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.169+0000 7f3f6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.169+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3f680038c0 con 0x7f3f7813f710 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.169+0000 7f3f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.169+0000 7f3f6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.169+0000 7f3f6e7fc640 10 monclient: dump: 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:43.171 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:43.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.169+0000 7f3f6ffff640 4 set_mon_vals no callback set 2024-03-25T18:22:43.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.170+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f3f68003aa0 con 0x7f3f7813f710 2024-03-25T18:22:43.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.171+0000 7f3f7d9fc640 10 monclient: get_auth_request con 0x7f3f5c06e0b0 auth_method 0 2024-03-25T18:22:43.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.171+0000 7f3f6e7fc640 10 client.4363.objecter ms_handle_connect 0x7f3f5c06e0b0 2024-03-25T18:22:43.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.172+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f3f68054710 con 0x7f3f7813f710 2024-03-25T18:22:43.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.172+0000 7f3f6e7fc640 10 client.4363.objecter ms_dispatch 0x7f3f78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:43.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.172+0000 7f3f6e7fc640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:43.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.172+0000 7f3f6e7fc640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:43.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.172+0000 7f3f6e7fc640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T18:22:43.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.451+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f68050050 con 0x7f3f7813f710 2024-03-25T18:22:43.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.451+0000 7f3f6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:43.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.451+0000 7f3f6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:43.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.709+0000 7f3f7fc60640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:43.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.709+0000 7f3f7fc60640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:43.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.709+0000 7f3f7fc60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:22:43.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.709+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f781386c0 con 0x7f3f7813f710 2024-03-25T18:22:43.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.732+0000 7f3f6e7fc640 1 -- 172.21.15.42:0/1636301052 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f3f68052150 con 0x7f3f7813f710 2024-03-25T18:22:43.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.732+0000 7f3f6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:43.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.732+0000 7f3f6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:43.734 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:43.734 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:43.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 10 librados: watch_flush enter 2024-03-25T18:22:43.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 10 librados: watch_flush exit 2024-03-25T18:22:43.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T18:22:43.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 10 client.4363.objecter successfully canceled tick 2024-03-25T18:22:43.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 >> v1:172.21.15.42:6800/3081750888 conn(0x7f3f5c06e0b0 legacy=0x7f3f5c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 10 monclient: shutdown 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.734+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 >> v1:172.21.15.120:6789/0 conn(0x7f3f7813f710 legacy=0x7f3f781da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.735+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 shutdown_connections 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.735+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 >> 172.21.15.42:0/1636301052 conn(0x7f3f780b7ea0 msgr2=0x7f3f78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.735+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 shutdown_connections 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.735+0000 7f3f7fc60640 1 -- 172.21.15.42:0/1636301052 wait complete. 2024-03-25T18:22:43.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:43.735+0000 7f3f7fc60640 1 librados: shutdown 2024-03-25T18:22:43.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:50.554 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.672+0000 7fc9af3c8640 10 monclient: get_monmap_and_config 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.672+0000 7fc9af3c8640 10 monclient: build_initial_monmap 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.672+0000 7fc9af3c8640 10 monclient: monmap: 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:50.673218+0000 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:50.673218+0000 2024-03-25T18:22:50.673 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.673+0000 7fc9af3c8640 1 Processor -- start 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.673+0000 7fc9af3c8640 1 -- start start 2024-03-25T18:22:50.674 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.673+0000 7fc9af3c8640 10 monclient: init 2024-03-25T18:22:50.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:50.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:50.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a80a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:50.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:50.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a81430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:50.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a8142520 con 0x7fc9a813f710 2024-03-25T18:22:50.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80b28a0 con 0x7fc9a80a98f0 2024-03-25T18:22:50.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80b2a30 con 0x7fc9a81430c0 2024-03-25T18:22:50.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:50.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.674+0000 7fc9af3c8640 10 monclient(hunting): authenticate will time out at 7485.901156s 2024-03-25T18:22:50.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.683+0000 7fc9ad965640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fc9a81430c0 0x7fc9a814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:34924/0 (socket says 172.21.15.42:34924) 2024-03-25T18:22:50.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.683+0000 7fc9ad965640 1 -- 172.21.15.42:0/4089442891 learned_addr learned my addr 172.21.15.42:0/4089442891 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:50.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.705+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921608414 0 0) 0x7fc9a8142520 con 0x7fc9a813f710 2024-03-25T18:22:50.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.705+0000 7fc99bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.705+0000 7fc99bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.705+0000 7fc99bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.705+0000 7fc99bfff640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:22:50.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.705+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc994003520 con 0x7fc9a813f710 2024-03-25T18:22:50.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.706+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164758392 0 0) 0x7fc9a80b2a30 con 0x7fc9a81430c0 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.706+0000 7fc99bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.706+0000 7fc99bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.706+0000 7fc99bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.706+0000 7fc99bfff640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.706+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a8142520 con 0x7fc9a81430c0 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.707+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840096055 0 0) 0x7fc9a80b28a0 con 0x7fc9a80a98f0 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.707+0000 7fc99bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.707+0000 7fc99bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.707+0000 7fc99bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.707+0000 7fc99bfff640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:22:50.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.707+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a80b2a30 con 0x7fc9a80a98f0 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3144528847 0 0) 0x7fc9a80b2a30 con 0x7fc9a80a98f0 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 >> v1:172.21.15.42:6790/0 conn(0x7fc9a81430c0 legacy=0x7fc9a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 >> v1:172.21.15.42:6789/0 conn(0x7fc9a813f710 legacy=0x7fc9a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 1 monclient: found mon.noname-c 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80b44c0 con 0x7fc9a80a98f0 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 10 monclient: _finish_auth 0 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc99c0015b0 con 0x7fc9a80a98f0 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:50.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc99bfff640 10 monclient: dump: 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc9af3c8640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:22:50.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.718+0000 7fc9af3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc99c001710 con 0x7fc9a80a98f0 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc99bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc99bfff640 1 -- 172.21.15.42:0/4089442891 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc99c001b60 con 0x7fc9a80a98f0 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc99bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc99bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc99bfff640 10 monclient: dump: 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.745 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc9af3c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:50.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.744+0000 7fc9af3c8640 4 set_mon_vals no callback set 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.745+0000 7fc9af3c8640 10 monclient: shutdown 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.745+0000 7fc9af3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.745+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4089442891 >> v1:172.21.15.120:6789/0 conn(0x7fc9a80a98f0 legacy=0x7fc9a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.745+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4089442891 shutdown_connections 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.745+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4089442891 >> 172.21.15.42:0/4089442891 conn(0x7fc9a80b7ea0 msgr2=0x7fc9a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.745+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4089442891 shutdown_connections 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.745+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4089442891 wait complete. 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 10 monclient: build_initial_monmap 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 10 monclient: monmap: 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:50.748 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 1 librados: starting msgr at 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 1 librados: starting objecter 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 1 Processor -- start 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 1 -- start start 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 1 librados: setting wanted keys 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 1 librados: calling monclient init 2024-03-25T18:22:50.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.746+0000 7fc9af3c8640 10 monclient: init 2024-03-25T18:22:50.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:50.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:50.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a80a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:50.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:50.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a81430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:50.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a81394b0 con 0x7fc9a81430c0 2024-03-25T18:22:50.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a8139720 con 0x7fc9a80a98f0 2024-03-25T18:22:50.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a814e760 con 0x7fc9a813f710 2024-03-25T18:22:50.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:50.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.747+0000 7fc9af3c8640 10 monclient(hunting): authenticate will time out at 7485.974239s 2024-03-25T18:22:50.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.752+0000 7fc9ad164640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc9a80a98f0 0x7fc9a81da980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:56288/0 (socket says 172.21.15.42:56288) 2024-03-25T18:22:50.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.752+0000 7fc9ad164640 1 -- 172.21.15.42:0/4215523065 learned_addr learned my addr 172.21.15.42:0/4215523065 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:50.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.756+0000 7fc999ffb640 10 client.?.objecter ms_handle_connect 0x7fc9a80a98f0 2024-03-25T18:22:50.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.756+0000 7fc999ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:50.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.761+0000 7fc999ffb640 10 client.?.objecter ms_handle_connect 0x7fc9a813f710 2024-03-25T18:22:50.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.761+0000 7fc999ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:50.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.772+0000 7fc999ffb640 10 client.?.objecter ms_handle_connect 0x7fc9a81430c0 2024-03-25T18:22:50.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.772+0000 7fc999ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:50.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980597105 0 0) 0x7fc9a8139720 con 0x7fc9a80a98f0 2024-03-25T18:22:50.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:22:50.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc98c004eb0 con 0x7fc9a80a98f0 2024-03-25T18:22:50.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766077553 0 0) 0x7fc9a814e760 con 0x7fc9a813f710 2024-03-25T18:22:50.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:22:50.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.773+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a8139720 con 0x7fc9a813f710 2024-03-25T18:22:50.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.779+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524939420 0 0) 0x7fc9a81394b0 con 0x7fc9a81430c0 2024-03-25T18:22:50.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.779+0000 7fc999ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.779+0000 7fc999ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.779+0000 7fc999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.779+0000 7fc999ffb640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:22:50.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.779+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a814e760 con 0x7fc9a81430c0 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.784+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889826994 0 0) 0x7fc9a8139720 con 0x7fc9a813f710 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.784+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a81394b0 con 0x7fc9a813f710 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.784+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc9a4001910 con 0x7fc9a813f710 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.784+0000 7fc999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.784+0000 7fc999ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.784+0000 7fc999ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:50.785 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:50.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.784+0000 7fc999ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.785+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512908163 0 0) 0x7fc98c004eb0 con 0x7fc9a80a98f0 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.785+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a8139720 con 0x7fc9a80a98f0 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.785+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc99c001b20 con 0x7fc9a80a98f0 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.785+0000 7fc999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.785+0000 7fc999ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.785+0000 7fc999ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:50.787 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:50.788 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:50.788 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:50.788 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:50.788 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:50.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1958245064 0 0) 0x7fc9a81394b0 con 0x7fc9a813f710 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 >> v1:172.21.15.120:6789/0 conn(0x7fc9a80a98f0 legacy=0x7fc9a81da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 >> v1:172.21.15.42:6789/0 conn(0x7fc9a81430c0 legacy=0x7fc9a81e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 1 monclient: found mon.c 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a814e8f0 con 0x7fc9a813f710 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc999ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 5 monclient: authenticate success, global_id 4367 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 10 monclient: _renew_subs 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:50.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a8137610 con 0x7fc9a813f710 2024-03-25T18:22:50.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:50.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 10 monclient: _renew_subs 2024-03-25T18:22:50.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:50.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a8138390 con 0x7fc9a813f710 2024-03-25T18:22:50.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.802+0000 7fc9af3c8640 1 librados: init done 2024-03-25T18:22:50.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.803+0000 7fc9af3c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:50.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.803+0000 7fc9af3c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:50.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.803+0000 7fc9af3c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:50.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:50.803+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a814f320 con 0x7fc9a813f710 2024-03-25T18:22:51.132 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.131+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9a4001d20 con 0x7fc9a813f710 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.131+0000 7fc999ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.131+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc9a40037b0 con 0x7fc9a813f710 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.131+0000 7fc999ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.131+0000 7fc999ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.131+0000 7fc999ffb640 10 monclient: dump: 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.131+0000 7fc99b7fe640 4 set_mon_vals no callback set 2024-03-25T18:22:51.134 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.132+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fc9a4019780 con 0x7fc9a813f710 2024-03-25T18:22:51.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.132+0000 7fc9ad164640 10 monclient: get_auth_request con 0x7fc98c06dec0 auth_method 0 2024-03-25T18:22:51.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.132+0000 7fc999ffb640 10 client.4367.objecter ms_handle_connect 0x7fc98c06dec0 2024-03-25T18:22:51.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fc9a4053520 con 0x7fc9a813f710 2024-03-25T18:22:51.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 10 client.4367.objecter ms_dispatch 0x7fc9a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:51.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:51.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:51.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-25T18:22:51.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9a404ee60 con 0x7fc9a813f710 2024-03-25T18:22:51.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:51.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.138+0000 7fc999ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:51.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.402+0000 7fc9af3c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:51.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.402+0000 7fc9af3c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:51.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.402+0000 7fc9af3c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:51.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.402+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9a80627b0 con 0x7fc9a813f710 2024-03-25T18:22:51.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.425+0000 7fc999ffb640 1 -- 172.21.15.42:0/4215523065 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fc9a4050f60 con 0x7fc9a813f710 2024-03-25T18:22:51.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.425+0000 7fc999ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:51.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.425+0000 7fc999ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:51.426 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:51.427 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:51.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 10 librados: watch_flush enter 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 10 librados: watch_flush exit 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 10 client.4367.objecter successfully canceled tick 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 >> v1:172.21.15.42:6800/3081750888 conn(0x7fc98c06dec0 legacy=0x7fc98c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 10 monclient: shutdown 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 >> v1:172.21.15.42:6790/0 conn(0x7fc9a813f710 legacy=0x7fc9a81db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 shutdown_connections 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 >> 172.21.15.42:0/4215523065 conn(0x7fc9a80b7ea0 msgr2=0x7fc9a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 shutdown_connections 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 1 -- 172.21.15.42:0/4215523065 wait complete. 2024-03-25T18:22:51.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:51.428+0000 7fc9af3c8640 1 librados: shutdown 2024-03-25T18:22:51.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:58.247 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:58.361 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.360+0000 7fbd1ef0e640 10 monclient: get_monmap_and_config 2024-03-25T18:22:58.361 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.360+0000 7fbd1ef0e640 10 monclient: build_initial_monmap 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.360+0000 7fbd1ef0e640 10 monclient: monmap: 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:22:58.361794+0000 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:22:58.361794+0000 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.361+0000 7fbd1ef0e640 1 Processor -- start 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.361+0000 7fbd1ef0e640 1 -- start start 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.361+0000 7fbd1ef0e640 10 monclient: init 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 10 monclient(hunting): picked mon.noname-c con 0x7fbd180a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 10 monclient(hunting): picked mon.noname-b con 0x7fbd181430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18142520 con 0x7fbd1813f710 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180b28a0 con 0x7fbd180a98f0 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180b2a30 con 0x7fbd181430c0 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.362+0000 7fbd1ef0e640 10 monclient(hunting): authenticate will time out at 7493.588859s 2024-03-25T18:22:58.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.375+0000 7fbd17fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbd1813f710 0x7fbd181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39728/0 (socket says 172.21.15.42:39728) 2024-03-25T18:22:58.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.375+0000 7fbd17fff640 1 -- 172.21.15.42:0/1597738796 learned_addr learned my addr 172.21.15.42:0/1597738796 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.395+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397699229 0 0) 0x7fbd180b28a0 con 0x7fbd180a98f0 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.395+0000 7fbd177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.395+0000 7fbd177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.395+0000 7fbd177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.395+0000 7fbd177fe640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.395+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd08003520 con 0x7fbd180a98f0 2024-03-25T18:22:58.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.400+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307929063 0 0) 0x7fbd180b2a30 con 0x7fbd181430c0 2024-03-25T18:22:58.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.400+0000 7fbd177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.400+0000 7fbd177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.400+0000 7fbd177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.400+0000 7fbd177fe640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:22:58.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.400+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd180b28a0 con 0x7fbd181430c0 2024-03-25T18:22:58.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.406+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632502405 0 0) 0x7fbd18142520 con 0x7fbd1813f710 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.406+0000 7fbd177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.406+0000 7fbd177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.406+0000 7fbd177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.406+0000 7fbd177fe640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.406+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd180b2a30 con 0x7fbd1813f710 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3709239413 0 0) 0x7fbd180b28a0 con 0x7fbd181430c0 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 >> v1:172.21.15.120:6789/0 conn(0x7fbd180a98f0 legacy=0x7fbd180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 >> v1:172.21.15.42:6789/0 conn(0x7fbd1813f710 legacy=0x7fbd181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 1 monclient: found mon.noname-b 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd180b44c0 con 0x7fbd181430c0 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:58.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbd100019f0 con 0x7fbd181430c0 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd177fe640 10 monclient: dump: 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:58.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd1ef0e640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:22:58.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.410+0000 7fbd1ef0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:58.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd10001b50 con 0x7fbd181430c0 2024-03-25T18:22:58.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:58.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd177fe640 1 -- 172.21.15.42:0/1597738796 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbd10001f60 con 0x7fbd181430c0 2024-03-25T18:22:58.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd177fe640 10 monclient: dump: 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:58.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd1ef0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:58.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.424+0000 7fbd1ef0e640 4 set_mon_vals no callback set 2024-03-25T18:22:58.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.425+0000 7fbd1ef0e640 10 monclient: shutdown 2024-03-25T18:22:58.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.425+0000 7fbd1ef0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:58.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.425+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/1597738796 >> v1:172.21.15.42:6790/0 conn(0x7fbd181430c0 legacy=0x7fbd1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.425+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/1597738796 shutdown_connections 2024-03-25T18:22:58.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.425+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/1597738796 >> 172.21.15.42:0/1597738796 conn(0x7fbd180b7ea0 msgr2=0x7fbd180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:58.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.425+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/1597738796 shutdown_connections 2024-03-25T18:22:58.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.425+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/1597738796 wait complete. 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 10 monclient: build_initial_monmap 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 10 monclient: monmap: 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 1 librados: starting msgr at 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 1 librados: starting objecter 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 1 Processor -- start 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 1 -- start start 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 1 librados: setting wanted keys 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 1 librados: calling monclient init 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.426+0000 7fbd1ef0e640 10 monclient: init 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 10 monclient(hunting): picked mon.noname-b con 0x7fbd180a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 10 monclient(hunting): picked mon.noname-c con 0x7fbd181430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180abac0 con 0x7fbd1813f710 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd181d4310 con 0x7fbd180a98f0 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18138f10 con 0x7fbd181430c0 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.427+0000 7fbd1ef0e640 10 monclient(hunting): authenticate will time out at 7493.654122s 2024-03-25T18:22:58.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.438+0000 7fbd1ccaa640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fbd180a98f0 0x7fbd18138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:42214/0 (socket says 172.21.15.42:42214) 2024-03-25T18:22:58.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.438+0000 7fbd1ccaa640 1 -- 172.21.15.42:0/909300253 learned_addr learned my addr 172.21.15.42:0/909300253 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:22:58.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.439+0000 7fbd157fa640 10 client.?.objecter ms_handle_connect 0x7fbd180a98f0 2024-03-25T18:22:58.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.439+0000 7fbd157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:58.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.445+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418623814 0 0) 0x7fbd181d4310 con 0x7fbd180a98f0 2024-03-25T18:22:58.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.445+0000 7fbd157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.445+0000 7fbd157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.445+0000 7fbd157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.445+0000 7fbd157fa640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:22:58.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.445+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcfc004eb0 con 0x7fbd180a98f0 2024-03-25T18:22:58.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.448+0000 7fbd157fa640 10 client.?.objecter ms_handle_connect 0x7fbd181430c0 2024-03-25T18:22:58.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.448+0000 7fbd157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:58.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.449+0000 7fbd157fa640 10 client.?.objecter ms_handle_connect 0x7fbd1813f710 2024-03-25T18:22:58.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.449+0000 7fbd157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:58.456 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.455+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015705933 0 0) 0x7fbd18138f10 con 0x7fbd181430c0 2024-03-25T18:22:58.456 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.455+0000 7fbd157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.456 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.455+0000 7fbd157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.456 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.455+0000 7fbd157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.455+0000 7fbd157fa640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:22:58.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.455+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd181d4310 con 0x7fbd181430c0 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 928129036 0 0) 0x7fbd181d4310 con 0x7fbd181430c0 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd18138f10 con 0x7fbd181430c0 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbd10001e60 con 0x7fbd181430c0 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 10 monclient(hunting): dump: 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.464 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1118464058 0 0) 0x7fbcfc004eb0 con 0x7fbd180a98f0 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.462+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd181d4310 con 0x7fbd180a98f0 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.463+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbd0c001990 con 0x7fbd180a98f0 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.463+0000 7fbd157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.463+0000 7fbd157fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.463+0000 7fbd157fa640 10 monclient(hunting): dump: 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:58.465 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:58.466 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:58.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.473+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057419127 0 0) 0x7fbd180abac0 con 0x7fbd1813f710 2024-03-25T18:22:58.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.473+0000 7fbd157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.473+0000 7fbd157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.473+0000 7fbd157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.473+0000 7fbd157fa640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:22:58.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.473+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcfc004eb0 con 0x7fbd1813f710 2024-03-25T18:22:58.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 892753547 0 0) 0x7fbd18138f10 con 0x7fbd181430c0 2024-03-25T18:22:58.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 >> v1:172.21.15.120:6789/0 conn(0x7fbd180a98f0 legacy=0x7fbd18138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 >> v1:172.21.15.42:6789/0 conn(0x7fbd1813f710 legacy=0x7fbd18134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 1 monclient: found mon.c 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18135010 con 0x7fbd181430c0 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 5 monclient: authenticate success, global_id 4373 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 10 monclient: _renew_subs 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:58.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd18134790 con 0x7fbd181430c0 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 10 monclient: _renew_subs 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd181d5120 con 0x7fbd181430c0 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.476+0000 7fbd1ef0e640 1 librados: init done 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.477+0000 7fbd1ef0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.477+0000 7fbd1ef0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.477+0000 7fbd1ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:58.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.477+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd181d44a0 con 0x7fbd181430c0 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.802+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd10002080 con 0x7fbd181430c0 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.802+0000 7fbd157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.802+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fbd10003dc0 con 0x7fbd181430c0 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.802+0000 7fbd157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.802+0000 7fbd157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.803+0000 7fbd157fa640 10 monclient: dump: 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.804 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:22:58.805 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.805 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:22:58.805 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:22:58.805 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:22:58.805 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:22:58.805 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:22:58.805 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.802+0000 7fbd16ffd640 4 set_mon_vals no callback set 2024-03-25T18:22:58.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.803+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fbd10019d90 con 0x7fbd181430c0 2024-03-25T18:22:58.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.804+0000 7fbd1ccaa640 10 monclient: get_auth_request con 0x7fbcfc06df00 auth_method 0 2024-03-25T18:22:58.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.804+0000 7fbd157fa640 10 client.4373.objecter ms_handle_connect 0x7fbcfc06df00 2024-03-25T18:22:58.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fbd10053930 con 0x7fbd181430c0 2024-03-25T18:22:58.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 10 client.4373.objecter ms_dispatch 0x7fbd18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:58.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:58.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:58.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-25T18:22:58.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd1004f440 con 0x7fbd181430c0 2024-03-25T18:22:58.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:58.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:58.811+0000 7fbd157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:59.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.067+0000 7fbd1ef0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:59.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.067+0000 7fbd1ef0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:59.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.067+0000 7fbd1ef0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:22:59.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.067+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd181351a0 con 0x7fbd181430c0 2024-03-25T18:22:59.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.087+0000 7fbd157fa640 1 -- 172.21.15.42:0/909300253 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fbd10051370 con 0x7fbd181430c0 2024-03-25T18:22:59.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.087+0000 7fbd157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:59.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.087+0000 7fbd157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:59.088 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:22:59.088 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 10 librados: watch_flush enter 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 10 librados: watch_flush exit 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 10 client.4373.objecter successfully canceled tick 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 >> v1:172.21.15.42:6800/3081750888 conn(0x7fbcfc06df00 legacy=0x7fbcfc0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 10 monclient: shutdown 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 >> v1:172.21.15.42:6790/0 conn(0x7fbd181430c0 legacy=0x7fbd181d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 shutdown_connections 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 >> 172.21.15.42:0/909300253 conn(0x7fbd180b7ea0 msgr2=0x7fbd18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 shutdown_connections 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.089+0000 7fbd1ef0e640 1 -- 172.21.15.42:0/909300253 wait complete. 2024-03-25T18:22:59.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:22:59.090+0000 7fbd1ef0e640 1 librados: shutdown 2024-03-25T18:22:59.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:01.709 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:44.878 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.767 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:44.936 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.802 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:03:44.971 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.834 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:44.954 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.838 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:44.958 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.840 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:45.009 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.855 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:45.025 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.870 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:03:44.990 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:01.891 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:03:45.061 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:05.909 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.023+0000 7fd333760640 10 monclient: get_monmap_and_config 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.023+0000 7fd333760640 10 monclient: build_initial_monmap 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.023+0000 7fd333760640 10 monclient: monmap: 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:06.024385+0000 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:06.024385+0000 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 1 Processor -- start 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 1 -- start start 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 10 monclient: init 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c13f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c1430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.024+0000 7fd333760640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c142520 con 0x7fd32c1430c0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.025+0000 7fd333760640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0b28a0 con 0x7fd32c13f710 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.025+0000 7fd333760640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0b2a30 con 0x7fd32c0a98f0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.025+0000 7fd333760640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.025+0000 7fd333760640 10 monclient(hunting): authenticate will time out at 7501.251669s 2024-03-25T18:23:06.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.045+0000 7fd331cfd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd32c1430c0 0x7fd32c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44446/0 (socket says 172.21.15.42:44446) 2024-03-25T18:23:06.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.045+0000 7fd331cfd640 1 -- 172.21.15.42:0/2288367708 learned_addr learned my addr 172.21.15.42:0/2288367708 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:06.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.057+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756825262 0 0) 0x7fd32c0b28a0 con 0x7fd32c13f710 2024-03-25T18:23:06.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.057+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.057+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.057+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.057+0000 7fd323fff640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:23:06.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.057+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd318003520 con 0x7fd32c13f710 2024-03-25T18:23:06.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910049219 0 0) 0x7fd32c0b2a30 con 0x7fd32c0a98f0 2024-03-25T18:23:06.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c0b28a0 con 0x7fd32c0a98f0 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340215087 0 0) 0x7fd32c142520 con 0x7fd32c1430c0 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:23:06.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.064+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c0b2a30 con 0x7fd32c1430c0 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058850731 0 0) 0x7fd318003520 con 0x7fd32c13f710 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 >> v1:172.21.15.42:6790/0 conn(0x7fd32c0a98f0 legacy=0x7fd32c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 >> v1:172.21.15.42:6789/0 conn(0x7fd32c1430c0 legacy=0x7fd32c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 1 monclient: found mon.noname-c 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0b44c0 con 0x7fd32c13f710 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd31c001690 con 0x7fd32c13f710 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd333760640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd333760640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd323fff640 10 monclient: dump: 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.071+0000 7fd333760640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:06.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd31c0017f0 con 0x7fd32c13f710 2024-03-25T18:23:06.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd323fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:06.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd323fff640 1 -- 172.21.15.42:0/2288367708 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd31c001c00 con 0x7fd32c13f710 2024-03-25T18:23:06.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd333760640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:06.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd333760640 4 set_mon_vals no callback set 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd323fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.085+0000 7fd323fff640 10 monclient: dump: 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:06.087 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.086+0000 7fd333760640 10 monclient: shutdown 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.086+0000 7fd333760640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.086+0000 7fd333760640 1 -- 172.21.15.42:0/2288367708 >> v1:172.21.15.120:6789/0 conn(0x7fd32c13f710 legacy=0x7fd32c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.086+0000 7fd333760640 1 -- 172.21.15.42:0/2288367708 shutdown_connections 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.086+0000 7fd333760640 1 -- 172.21.15.42:0/2288367708 >> 172.21.15.42:0/2288367708 conn(0x7fd32c0b7ea0 msgr2=0x7fd32c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.087+0000 7fd333760640 1 -- 172.21.15.42:0/2288367708 shutdown_connections 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.087+0000 7fd333760640 1 -- 172.21.15.42:0/2288367708 wait complete. 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.088+0000 7fd333760640 10 monclient: build_initial_monmap 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.088+0000 7fd333760640 10 monclient: monmap: 2024-03-25T18:23:06.089 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:06.090 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.088+0000 7fd333760640 1 librados: starting msgr at 2024-03-25T18:23:06.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.088+0000 7fd333760640 1 librados: starting objecter 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.088+0000 7fd333760640 1 Processor -- start 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 1 -- start start 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 1 librados: setting wanted keys 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 1 librados: calling monclient init 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient: init 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c0a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c13f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c06eec0 con 0x7fd32c0a98f0 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c133720 con 0x7fd32c1430c0 2024-03-25T18:23:06.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c06f130 con 0x7fd32c13f710 2024-03-25T18:23:06.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:06.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.089+0000 7fd333760640 10 monclient(hunting): authenticate will time out at 7501.316275s 2024-03-25T18:23:06.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.102+0000 7fd330cfb640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fd32c13f710 0x7fd32c132900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:37576/0 (socket says 172.21.15.42:37576) 2024-03-25T18:23:06.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.102+0000 7fd330cfb640 1 -- 172.21.15.42:0/4258178697 learned_addr learned my addr 172.21.15.42:0/4258178697 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:06.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.105+0000 7fd321ffb640 10 client.?.objecter ms_handle_connect 0x7fd32c13f710 2024-03-25T18:23:06.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.105+0000 7fd321ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:06.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.106+0000 7fd321ffb640 10 client.?.objecter ms_handle_connect 0x7fd32c1430c0 2024-03-25T18:23:06.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.106+0000 7fd321ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:06.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.107+0000 7fd321ffb640 10 client.?.objecter ms_handle_connect 0x7fd32c0a98f0 2024-03-25T18:23:06.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.107+0000 7fd321ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:06.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.119+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194163480 0 0) 0x7fd32c133720 con 0x7fd32c1430c0 2024-03-25T18:23:06.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.119+0000 7fd321ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.119+0000 7fd321ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.119+0000 7fd321ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.119+0000 7fd321ffb640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:23:06.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.119+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd310004eb0 con 0x7fd32c1430c0 2024-03-25T18:23:06.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.122+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999537182 0 0) 0x7fd32c06f130 con 0x7fd32c13f710 2024-03-25T18:23:06.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.122+0000 7fd321ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.122+0000 7fd321ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.122+0000 7fd321ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.122+0000 7fd321ffb640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:23:06.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.122+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c133720 con 0x7fd32c13f710 2024-03-25T18:23:06.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.124+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646521239 0 0) 0x7fd32c06eec0 con 0x7fd32c0a98f0 2024-03-25T18:23:06.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.124+0000 7fd321ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.124+0000 7fd321ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.124+0000 7fd321ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.124+0000 7fd321ffb640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:23:06.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.124+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c06f130 con 0x7fd32c0a98f0 2024-03-25T18:23:06.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.134+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980798660 0 0) 0x7fd32c133720 con 0x7fd32c13f710 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.134+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c06eec0 con 0x7fd32c13f710 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.134+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd31c001b40 con 0x7fd32c13f710 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.134+0000 7fd321ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:06.136 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:06.137 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:06.137 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4248627005 0 0) 0x7fd310004eb0 con 0x7fd32c1430c0 2024-03-25T18:23:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c133720 con 0x7fd32c1430c0 2024-03-25T18:23:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd324001bd0 con 0x7fd32c1430c0 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.135+0000 7fd321ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:06.138 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.145+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2235789593 0 0) 0x7fd32c06f130 con 0x7fd32c0a98f0 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.145+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd310004eb0 con 0x7fd32c0a98f0 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.145+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd3280018f0 con 0x7fd32c0a98f0 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.145+0000 7fd321ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.145+0000 7fd321ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.145+0000 7fd321ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:06.146 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:06.147 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 607548016 0 0) 0x7fd32c133720 con 0x7fd32c1430c0 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 >> v1:172.21.15.42:6790/0 conn(0x7fd32c13f710 legacy=0x7fd32c132900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 >> v1:172.21.15.42:6789/0 conn(0x7fd32c0a98f0 legacy=0x7fd32c06f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 1 monclient: found mon.b 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:06.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c06f410 con 0x7fd32c1430c0 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd321ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd333760640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd333760640 10 monclient: _renew_subs 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd333760640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.147+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c14da00 con 0x7fd32c1430c0 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.148+0000 7fd333760640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.148+0000 7fd333760640 10 monclient: _renew_subs 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.148+0000 7fd333760640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.148+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c14e300 con 0x7fd32c1430c0 2024-03-25T18:23:06.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.148+0000 7fd333760640 1 librados: init done 2024-03-25T18:23:06.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.149+0000 7fd333760640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:06.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.149+0000 7fd333760640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:06.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.149+0000 7fd333760640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:06.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.149+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c0627b0 con 0x7fd32c1430c0 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.432+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd324002060 con 0x7fd32c1430c0 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.432+0000 7fd321ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.432+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd324004d00 con 0x7fd32c1430c0 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.432+0000 7fd321ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.432+0000 7fd321ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.432+0000 7fd321ffb640 10 monclient: dump: 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:06.434 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:06.435 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.432+0000 7fd3237fe640 4 set_mon_vals no callback set 2024-03-25T18:23:06.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.433+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fd32401acd0 con 0x7fd32c1430c0 2024-03-25T18:23:06.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.434+0000 7fd3314fc640 10 monclient: get_auth_request con 0x7fd31006e110 auth_method 0 2024-03-25T18:23:06.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.434+0000 7fd321ffb640 10 client.4381.objecter ms_handle_connect 0x7fd31006e110 2024-03-25T18:23:06.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.438+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fd3240549c0 con 0x7fd32c1430c0 2024-03-25T18:23:06.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.438+0000 7fd321ffb640 10 client.4381.objecter ms_dispatch 0x7fd32c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:06.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.438+0000 7fd321ffb640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:06.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.438+0000 7fd321ffb640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:06.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.438+0000 7fd321ffb640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T18:23:06.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.439+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3240504d0 con 0x7fd32c1430c0 2024-03-25T18:23:06.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.439+0000 7fd321ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:06.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.439+0000 7fd321ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:06.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.705+0000 7fd333760640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:06.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.705+0000 7fd333760640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:06.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.705+0000 7fd333760640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:06.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.705+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd32c13e250 con 0x7fd32c1430c0 2024-03-25T18:23:06.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.729+0000 7fd321ffb640 1 -- 172.21.15.42:0/4258178697 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fd324052400 con 0x7fd32c1430c0 2024-03-25T18:23:06.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.729+0000 7fd321ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:06.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.729+0000 7fd321ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:06.731 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:23:06.731 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:06.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.731+0000 7fd333760640 10 librados: watch_flush enter 2024-03-25T18:23:06.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 10 librados: watch_flush exit 2024-03-25T18:23:06.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T18:23:06.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 10 client.4381.objecter successfully canceled tick 2024-03-25T18:23:06.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 >> v1:172.21.15.42:6800/3081750888 conn(0x7fd31006e110 legacy=0x7fd3100705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 10 monclient: shutdown 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 >> v1:172.21.15.120:6789/0 conn(0x7fd32c1430c0 legacy=0x7fd32c133010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 shutdown_connections 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 >> 172.21.15.42:0/4258178697 conn(0x7fd32c0b7ea0 msgr2=0x7fd32c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 shutdown_connections 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 1 -- 172.21.15.42:0/4258178697 wait complete. 2024-03-25T18:23:06.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:06.732+0000 7fd333760640 1 librados: shutdown 2024-03-25T18:23:06.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:13.550 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:13.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.663+0000 7f90cdb82640 10 monclient: get_monmap_and_config 2024-03-25T18:23:13.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.663+0000 7f90cdb82640 10 monclient: build_initial_monmap 2024-03-25T18:23:13.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.663+0000 7f90cdb82640 10 monclient: monmap: 2024-03-25T18:23:13.664 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:13.664 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:13.664 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:13.664266+0000 2024-03-25T18:23:13.664 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:13.664266+0000 2024-03-25T18:23:13.665 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:13.665 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:13.665 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:13.665 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:13.665 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:13.665 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:13.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.663+0000 7f90cdb82640 1 Processor -- start 2024-03-25T18:23:13.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 1 -- start start 2024-03-25T18:23:13.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient: init 2024-03-25T18:23:13.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:13.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient(hunting): picked mon.noname-a con 0x7f90c80b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient(hunting): picked mon.noname-b con 0x7f90c80b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient(hunting): picked mon.noname-c con 0x7f90c814be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c80af940 con 0x7f90c80b4d60 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c80b0a70 con 0x7f90c814be90 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c80b0d50 con 0x7f90c80b1730 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:13.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.664+0000 7f90cdb82640 10 monclient(hunting): authenticate will time out at 7508.891300s 2024-03-25T18:23:13.674 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.673+0000 7f90c77fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f90c80b4d60 0x7f90c813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34626/0 (socket says 172.21.15.42:34626) 2024-03-25T18:23:13.674 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.673+0000 7f90c77fe640 1 -- 172.21.15.42:0/884083165 learned_addr learned my addr 172.21.15.42:0/884083165 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:13.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.684+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150270032 0 0) 0x7f90c80af940 con 0x7f90c80b4d60 2024-03-25T18:23:13.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.684+0000 7f90c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.684+0000 7f90c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.684+0000 7f90c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.684+0000 7f90c67fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:23:13.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.684+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b0003520 con 0x7f90c80b4d60 2024-03-25T18:23:13.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.687+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263913525 0 0) 0x7f90c80b0a70 con 0x7f90c814be90 2024-03-25T18:23:13.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.687+0000 7f90c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.687+0000 7f90c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.687+0000 7f90c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.687+0000 7f90c67fc640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:23:13.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.687+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c80af940 con 0x7f90c814be90 2024-03-25T18:23:13.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.697+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246946034 0 0) 0x7f90c80b0d50 con 0x7f90c80b1730 2024-03-25T18:23:13.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.697+0000 7f90c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.697+0000 7f90c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.697+0000 7f90c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.697+0000 7f90c67fc640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:23:13.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.697+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c80b0a70 con 0x7f90c80b1730 2024-03-25T18:23:13.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.708+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394350631 0 0) 0x7f90c80af940 con 0x7f90c814be90 2024-03-25T18:23:13.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.708+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 >> v1:172.21.15.42:6790/0 conn(0x7f90c80b1730 legacy=0x7f90c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.708+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 >> v1:172.21.15.42:6789/0 conn(0x7f90c80b4d60 legacy=0x7f90c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.708+0000 7f90c67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.708+0000 7f90c67fc640 1 monclient: found mon.noname-c 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.708+0000 7f90c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.708+0000 7f90c67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c80b1030 con 0x7f90c814be90 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f90c00019f0 con 0x7f90c814be90 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90cdb82640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90cdb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:13.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90c67fc640 10 monclient: dump: 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:13.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.709+0000 7f90cdb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:13.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90c0001b50 con 0x7f90c814be90 2024-03-25T18:23:13.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:13.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90c67fc640 1 -- 172.21.15.42:0/884083165 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f90c0001f60 con 0x7f90c814be90 2024-03-25T18:23:13.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:13.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90c67fc640 10 monclient: dump: 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90cdb82640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:13.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.730+0000 7f90cdb82640 4 set_mon_vals no callback set 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.731+0000 7f90cdb82640 10 monclient: shutdown 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.731+0000 7f90cdb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.731+0000 7f90cdb82640 1 -- 172.21.15.42:0/884083165 >> v1:172.21.15.120:6789/0 conn(0x7f90c814be90 legacy=0x7f90c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.731+0000 7f90cdb82640 1 -- 172.21.15.42:0/884083165 shutdown_connections 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.731+0000 7f90cdb82640 1 -- 172.21.15.42:0/884083165 >> 172.21.15.42:0/884083165 conn(0x7f90c80617a0 msgr2=0x7f90c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.731+0000 7f90cdb82640 1 -- 172.21.15.42:0/884083165 shutdown_connections 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.731+0000 7f90cdb82640 1 -- 172.21.15.42:0/884083165 wait complete. 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.732+0000 7f90cdb82640 10 monclient: build_initial_monmap 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.732+0000 7f90cdb82640 10 monclient: monmap: 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.733+0000 7f90cdb82640 1 librados: starting msgr at 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.733+0000 7f90cdb82640 1 librados: starting objecter 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.733+0000 7f90cdb82640 1 Processor -- start 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.733+0000 7f90cdb82640 1 -- start start 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.733+0000 7f90cdb82640 1 librados: setting wanted keys 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.733+0000 7f90cdb82640 1 librados: calling monclient init 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.733+0000 7f90cdb82640 10 monclient: init 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 10 monclient(hunting): picked mon.noname-b con 0x7f90c80b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 10 monclient(hunting): picked mon.noname-c con 0x7f90c80b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 10 monclient(hunting): picked mon.noname-a con 0x7f90c814be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c80aabb0 con 0x7f90c814be90 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c80a95a0 con 0x7f90c80b1730 2024-03-25T18:23:13.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c813b540 con 0x7f90c80b4d60 2024-03-25T18:23:13.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:13.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.734+0000 7f90cdb82640 10 monclient(hunting): authenticate will time out at 7508.960955s 2024-03-25T18:23:13.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.741+0000 7f90c7fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f90c814be90 0x7f90c81d8120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34664/0 (socket says 172.21.15.42:34664) 2024-03-25T18:23:13.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.741+0000 7f90c7fff640 1 -- 172.21.15.42:0/484942789 learned_addr learned my addr 172.21.15.42:0/484942789 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:13.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.744+0000 7f90affff640 10 client.?.objecter ms_handle_connect 0x7f90c814be90 2024-03-25T18:23:13.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.744+0000 7f90affff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:13.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.748+0000 7f90affff640 10 client.?.objecter ms_handle_connect 0x7f90c80b4d60 2024-03-25T18:23:13.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.748+0000 7f90affff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:13.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.751+0000 7f90affff640 10 client.?.objecter ms_handle_connect 0x7f90c80b1730 2024-03-25T18:23:13.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.751+0000 7f90affff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:13.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.756+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336044451 0 0) 0x7f90c80aabb0 con 0x7f90c814be90 2024-03-25T18:23:13.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.756+0000 7f90affff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.756+0000 7f90affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.756+0000 7f90affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.756+0000 7f90affff640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:23:13.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.756+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4004eb0 con 0x7f90c814be90 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.764+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3018308000 0 0) 0x7f90b4004eb0 con 0x7f90c814be90 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.764+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90c80aabb0 con 0x7f90c814be90 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.764+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f90c0001fa0 con 0x7f90c814be90 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.764+0000 7f90affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.764+0000 7f90affff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.764+0000 7f90affff640 10 monclient(hunting): dump: 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.764+0000 7f90affff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3252799674 0 0) 0x7f90c80aabb0 con 0x7f90c814be90 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 >> v1:172.21.15.42:6790/0 conn(0x7f90c80b4d60 legacy=0x7f90c81d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 >> v1:172.21.15.120:6789/0 conn(0x7f90c80b1730 legacy=0x7f90c8137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 1 monclient: found mon.a 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c813b6d0 con 0x7f90c814be90 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 10 monclient: _finish_auth 0 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:13.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 10 monclient: _renew_subs 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c8138680 con 0x7f90c814be90 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 10 monclient: _renew_subs 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c8138f40 con 0x7f90c814be90 2024-03-25T18:23:13.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.769+0000 7f90cdb82640 1 librados: init done 2024-03-25T18:23:13.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.770+0000 7f90cdb82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:13.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.770+0000 7f90cdb82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:13.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.770+0000 7f90cdb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:13.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:13.770+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c8138980 con 0x7f90c814be90 2024-03-25T18:23:14.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.082+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90c0003c20 con 0x7f90c814be90 2024-03-25T18:23:14.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.082+0000 7f90affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:14.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.082+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f90c0004030 con 0x7f90c814be90 2024-03-25T18:23:14.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.082+0000 7f90affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.082+0000 7f90affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.082+0000 7f90affff640 10 monclient: dump: 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:14.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.082+0000 7f90c5ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:14.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.083+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f90c0004250 con 0x7f90c814be90 2024-03-25T18:23:14.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.083+0000 7f90c77fe640 10 monclient: get_auth_request con 0x7f90b406dc70 auth_method 0 2024-03-25T18:23:14.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.084+0000 7f90affff640 10 client.4386.objecter ms_handle_connect 0x7f90b406dc70 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.089+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f90c0054b40 con 0x7f90c814be90 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.089+0000 7f90affff640 10 client.4386.objecter ms_dispatch 0x7f90c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.089+0000 7f90affff640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.089+0000 7f90affff640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.090+0000 7f90affff640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.090+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90c0050650 con 0x7f90c814be90 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.090+0000 7f90affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:14.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.090+0000 7f90affff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:14.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.362+0000 7f90cdb82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:14.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.362+0000 7f90cdb82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:14.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.362+0000 7f90cdb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:14.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.362+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90c80afbd0 con 0x7f90c814be90 2024-03-25T18:23:14.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.409+0000 7f90affff640 1 -- 172.21.15.42:0/484942789 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f90c0052580 con 0x7f90c814be90 2024-03-25T18:23:14.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.409+0000 7f90affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:14.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.409+0000 7f90affff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:14.411 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:23:14.411 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:14.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.411+0000 7f90cdb82640 10 librados: watch_flush enter 2024-03-25T18:23:14.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 10 librados: watch_flush exit 2024-03-25T18:23:14.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:23:14.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 >> v1:172.21.15.42:6800/3081750888 conn(0x7f90b406dc70 legacy=0x7f90b4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 10 monclient: shutdown 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 >> v1:172.21.15.42:6789/0 conn(0x7f90c814be90 legacy=0x7f90c81d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 shutdown_connections 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 >> 172.21.15.42:0/484942789 conn(0x7f90c80617a0 msgr2=0x7f90c813beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 shutdown_connections 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 1 -- 172.21.15.42:0/484942789 wait complete. 2024-03-25T18:23:14.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:14.412+0000 7f90cdb82640 1 librados: shutdown 2024-03-25T18:23:14.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:21.230 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:21.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.347+0000 7fb733d5d640 10 monclient: get_monmap_and_config 2024-03-25T18:23:21.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.347+0000 7fb733d5d640 10 monclient: build_initial_monmap 2024-03-25T18:23:21.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.347+0000 7fb733d5d640 10 monclient: monmap: 2024-03-25T18:23:21.348 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:21.348575+0000 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:21.348575+0000 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:21.349 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:21.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.348+0000 7fb733d5d640 1 Processor -- start 2024-03-25T18:23:21.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.348+0000 7fb733d5d640 1 -- start start 2024-03-25T18:23:21.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.348+0000 7fb733d5d640 10 monclient: init 2024-03-25T18:23:21.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb72c0b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb72c0b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb72c14be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0af940 con 0x7fb72c14be90 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0b0a70 con 0x7fb72c0b4d60 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0b0d50 con 0x7fb72c0b1730 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:21.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.349+0000 7fb733d5d640 10 monclient(hunting): authenticate will time out at 7516.575994s 2024-03-25T18:23:21.358 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.357+0000 7fb7322fa640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb72c14be90 0x7fb72c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43926/0 (socket says 172.21.15.42:43926) 2024-03-25T18:23:21.358 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.357+0000 7fb7322fa640 1 -- 172.21.15.42:0/2553276425 learned_addr learned my addr 172.21.15.42:0/2553276425 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.373+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105668632 0 0) 0x7fb72c0af940 con 0x7fb72c14be90 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.373+0000 7fb730af7640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.373+0000 7fb730af7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.373+0000 7fb730af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.373+0000 7fb730af7640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.373+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb718003520 con 0x7fb72c14be90 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.375+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230851974 0 0) 0x7fb72c0b0a70 con 0x7fb72c0b4d60 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.375+0000 7fb730af7640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.375+0000 7fb730af7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.375+0000 7fb730af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.375+0000 7fb730af7640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:23:21.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.375+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c0af940 con 0x7fb72c0b4d60 2024-03-25T18:23:21.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.382+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577534438 0 0) 0x7fb72c0b0d50 con 0x7fb72c0b1730 2024-03-25T18:23:21.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.382+0000 7fb730af7640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.382+0000 7fb730af7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.382+0000 7fb730af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.382+0000 7fb730af7640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:23:21.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.382+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c0b0a70 con 0x7fb72c0b1730 2024-03-25T18:23:21.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161725022 0 0) 0x7fb72c0b0a70 con 0x7fb72c0b1730 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 >> v1:172.21.15.120:6789/0 conn(0x7fb72c0b4d60 legacy=0x7fb72c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 >> v1:172.21.15.42:6789/0 conn(0x7fb72c14be90 legacy=0x7fb72c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 10 monclient: _finish_hunting 0 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 1 monclient: found mon.noname-b 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0b1030 con 0x7fb72c0b1730 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 10 monclient: _finish_auth 0 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 10 monclient: _check_auth_tickets 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb733d5d640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb71c0015b0 con 0x7fb72c0b1730 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb733d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb730af7640 10 monclient: dump: 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:21.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.390+0000 7fb733d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.397+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb71c001710 con 0x7fb72c0b1730 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.397+0000 7fb730af7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.397+0000 7fb730af7640 1 -- 172.21.15.42:0/2553276425 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb71c001b60 con 0x7fb72c0b1730 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb730af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb730af7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb730af7640 10 monclient: dump: 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:21.399 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:21.400 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:21.400 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:21.400 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:21.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb733d5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:21.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb733d5d640 4 set_mon_vals no callback set 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb733d5d640 10 monclient: shutdown 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb733d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb733d5d640 1 -- 172.21.15.42:0/2553276425 >> v1:172.21.15.42:6790/0 conn(0x7fb72c0b1730 legacy=0x7fb72c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.398+0000 7fb733d5d640 1 -- 172.21.15.42:0/2553276425 shutdown_connections 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.399+0000 7fb733d5d640 1 -- 172.21.15.42:0/2553276425 >> 172.21.15.42:0/2553276425 conn(0x7fb72c0617a0 msgr2=0x7fb72c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.399+0000 7fb733d5d640 1 -- 172.21.15.42:0/2553276425 shutdown_connections 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.399+0000 7fb733d5d640 1 -- 172.21.15.42:0/2553276425 wait complete. 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.399+0000 7fb733d5d640 10 monclient: build_initial_monmap 2024-03-25T18:23:21.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.399+0000 7fb733d5d640 10 monclient: monmap: 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 librados: starting msgr at 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 librados: starting objecter 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 Processor -- start 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 -- start start 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 librados: setting wanted keys 2024-03-25T18:23:21.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 librados: calling monclient init 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient: init 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb72c0b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb72c0b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb72c14be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c138420 con 0x7fb72c0b1730 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c1d77a0 con 0x7fb72c14be90 2024-03-25T18:23:21.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c1d83d0 con 0x7fb72c0b4d60 2024-03-25T18:23:21.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:21.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.400+0000 7fb733d5d640 10 monclient(hunting): authenticate will time out at 7516.627544s 2024-03-25T18:23:21.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.415+0000 7fb731af9640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb72c0b1730 0x7fb72c1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43966/0 (socket says 172.21.15.42:43966) 2024-03-25T18:23:21.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.415+0000 7fb731af9640 1 -- 172.21.15.42:0/1021923769 learned_addr learned my addr 172.21.15.42:0/1021923769 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:21.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.417+0000 7fb7227fc640 10 client.?.objecter ms_handle_connect 0x7fb72c0b1730 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.417+0000 7fb7227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:21.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.426+0000 7fb7227fc640 10 client.?.objecter ms_handle_connect 0x7fb72c0b4d60 2024-03-25T18:23:21.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.426+0000 7fb7227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:21.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.427+0000 7fb7227fc640 10 client.?.objecter ms_handle_connect 0x7fb72c14be90 2024-03-25T18:23:21.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.427+0000 7fb7227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:21.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.428+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839712540 0 0) 0x7fb72c138420 con 0x7fb72c0b1730 2024-03-25T18:23:21.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.428+0000 7fb7227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.428+0000 7fb7227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.428+0000 7fb7227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.428+0000 7fb7227fc640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:23:21.429 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.428+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb710004eb0 con 0x7fb72c0b1730 2024-03-25T18:23:21.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.444+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832963421 0 0) 0x7fb72c1d77a0 con 0x7fb72c14be90 2024-03-25T18:23:21.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.444+0000 7fb7227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.444+0000 7fb7227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.444+0000 7fb7227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.444+0000 7fb7227fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:23:21.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.444+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c138420 con 0x7fb72c14be90 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086802726 0 0) 0x7fb72c1d83d0 con 0x7fb72c0b4d60 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c1d77a0 con 0x7fb72c0b4d60 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1050197098 0 0) 0x7fb710004eb0 con 0x7fb72c0b1730 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb72c1d83d0 con 0x7fb72c0b1730 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb728001a10 con 0x7fb72c0b1730 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): dump: 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:21.447 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:21.448 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.445+0000 7fb7227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:21.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.458+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2962502679 0 0) 0x7fb72c138420 con 0x7fb72c14be90 2024-03-25T18:23:21.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.458+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb710004eb0 con 0x7fb72c14be90 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.458+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb724001ab0 con 0x7fb72c14be90 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.458+0000 7fb7227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.458+0000 7fb7227fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.458+0000 7fb7227fc640 10 monclient(hunting): dump: 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:21.460 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:21.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 270546274 0 0) 0x7fb72c1d83d0 con 0x7fb72c0b1730 2024-03-25T18:23:21.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 >> v1:172.21.15.42:6790/0 conn(0x7fb72c0b4d60 legacy=0x7fb72c136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 >> v1:172.21.15.120:6789/0 conn(0x7fb72c14be90 legacy=0x7fb72c13b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:21.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 1 monclient: found mon.a 2024-03-25T18:23:21.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:21.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c135f80 con 0x7fb72c0b1730 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb7227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 10 monclient: _renew_subs 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c137230 con 0x7fb72c0b1730 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 10 monclient: _renew_subs 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c143ae0 con 0x7fb72c0b1730 2024-03-25T18:23:21.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.463+0000 7fb733d5d640 1 librados: init done 2024-03-25T18:23:21.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.464+0000 7fb733d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:21.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.464+0000 7fb733d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:21.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.464+0000 7fb733d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:21.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.464+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c136170 con 0x7fb72c0b1730 2024-03-25T18:23:21.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb728001c50 con 0x7fb72c0b1730 2024-03-25T18:23:21.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb7227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:21.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb728003a60 con 0x7fb72c0b1730 2024-03-25T18:23:21.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb7227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb7227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:21.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb7227fc640 10 monclient: dump: 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:21.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb723fff640 4 set_mon_vals no callback set 2024-03-25T18:23:21.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.807+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fb728003c80 con 0x7fb72c0b1730 2024-03-25T18:23:21.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.808+0000 7fb7312f8640 10 monclient: get_auth_request con 0x7fb71006df70 auth_method 0 2024-03-25T18:23:21.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.808+0000 7fb7227fc640 10 client.4392.objecter ms_handle_connect 0x7fb71006df70 2024-03-25T18:23:21.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fb7280548d0 con 0x7fb72c0b1730 2024-03-25T18:23:21.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 10 client.4392.objecter ms_dispatch 0x7fb72c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:21.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:21.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:21.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-25T18:23:21.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7280503e0 con 0x7fb72c0b1730 2024-03-25T18:23:21.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:21.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:21.813+0000 7fb7227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:21.909 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:04:05.078 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:21.920 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:04:05.090 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:21.970 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:04:05.139 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:21.986 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:04:05.156 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:21.997 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:04:05.166 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:22.015 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:04:05.136 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:22.020 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:04:05.190 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:22.021 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:04:05.141 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:22.036 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.035+0000 7fb733d5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:22.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.035+0000 7fb733d5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:22.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.035+0000 7fb733d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:22.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.035+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb72c0627b0 con 0x7fb72c0b1730 2024-03-25T18:23:22.046 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:04:05.166 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:22.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.193+0000 7fb7227fc640 1 -- 172.21.15.42:0/1021923769 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fb728052310 con 0x7fb72c0b1730 2024-03-25T18:23:22.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.193+0000 7fb7227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:22.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.193+0000 7fb7227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:22.194 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:23:22.195 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:22.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 10 librados: watch_flush enter 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 10 librados: watch_flush exit 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 10 client.4392.objecter successfully canceled tick 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 >> v1:172.21.15.42:6800/3081750888 conn(0x7fb71006df70 legacy=0x7fb710070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 10 monclient: shutdown 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 >> v1:172.21.15.42:6789/0 conn(0x7fb72c0b1730 legacy=0x7fb72c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 shutdown_connections 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 >> 172.21.15.42:0/1021923769 conn(0x7fb72c0617a0 msgr2=0x7fb72c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 shutdown_connections 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 1 -- 172.21.15.42:0/1021923769 wait complete. 2024-03-25T18:23:22.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:22.195+0000 7fb733d5d640 1 librados: shutdown 2024-03-25T18:23:22.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:29.014 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:29.134 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.134+0000 7f2f278e4640 10 monclient: get_monmap_and_config 2024-03-25T18:23:29.134 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.134+0000 7f2f278e4640 10 monclient: build_initial_monmap 2024-03-25T18:23:29.134 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.134+0000 7f2f278e4640 10 monclient: monmap: 2024-03-25T18:23:29.134 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:29.134152+0000 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:29.134152+0000 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:29.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.134+0000 7f2f278e4640 1 Processor -- start 2024-03-25T18:23:29.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 1 -- start start 2024-03-25T18:23:29.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient: init 2024-03-25T18:23:29.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f200a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f2013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f201430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20142520 con 0x7f2f200a98f0 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f200b28a0 con 0x7f2f201430c0 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f200b2a30 con 0x7f2f2013f710 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:29.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.135+0000 7f2f278e4640 10 monclient(hunting): authenticate will time out at 7524.361518s 2024-03-25T18:23:29.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.143+0000 7f2f25e81640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2f201430c0 0x7f2f2014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34054/0 (socket says 172.21.15.42:34054) 2024-03-25T18:23:29.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.143+0000 7f2f25e81640 1 -- 172.21.15.42:0/2945621403 learned_addr learned my addr 172.21.15.42:0/2945621403 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:29.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.161+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034346211 0 0) 0x7f2f20142520 con 0x7f2f200a98f0 2024-03-25T18:23:29.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.161+0000 7f2f17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.161+0000 7f2f17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.161+0000 7f2f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.161+0000 7f2f17fff640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:23:29.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.161+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c003520 con 0x7f2f200a98f0 2024-03-25T18:23:29.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020601002 0 0) 0x7f2f200b28a0 con 0x7f2f201430c0 2024-03-25T18:23:29.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:23:29.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f20142520 con 0x7f2f201430c0 2024-03-25T18:23:29.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262491725 0 0) 0x7f2f200b2a30 con 0x7f2f2013f710 2024-03-25T18:23:29.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:23:29.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.163+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f200b28a0 con 0x7f2f2013f710 2024-03-25T18:23:29.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725338537 0 0) 0x7f2f200b28a0 con 0x7f2f2013f710 2024-03-25T18:23:29.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 >> v1:172.21.15.120:6789/0 conn(0x7f2f201430c0 legacy=0x7f2f2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 >> v1:172.21.15.42:6789/0 conn(0x7f2f200a98f0 legacy=0x7f2f200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 1 monclient: found mon.noname-b 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f200b44c0 con 0x7f2f2013f710 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f100015b0 con 0x7f2f2013f710 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f17fff640 10 monclient: dump: 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:29.182 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f278e4640 5 monclient: authenticate success, global_id 4394 2024-03-25T18:23:29.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.181+0000 7f2f278e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:29.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f10001710 con 0x7f2f2013f710 2024-03-25T18:23:29.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f17fff640 1 -- 172.21.15.42:0/2945621403 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f10001b60 con 0x7f2f2013f710 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f278e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f278e4640 4 set_mon_vals no callback set 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.196+0000 7f2f17fff640 10 monclient: dump: 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:29.197 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:29.198 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:29.198 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:29.198 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.197+0000 7f2f278e4640 10 monclient: shutdown 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.197+0000 7f2f278e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.197+0000 7f2f278e4640 1 -- 172.21.15.42:0/2945621403 >> v1:172.21.15.42:6790/0 conn(0x7f2f2013f710 legacy=0x7f2f201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.197+0000 7f2f278e4640 1 -- 172.21.15.42:0/2945621403 shutdown_connections 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.197+0000 7f2f278e4640 1 -- 172.21.15.42:0/2945621403 >> 172.21.15.42:0/2945621403 conn(0x7f2f200b7ea0 msgr2=0x7f2f200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.198+0000 7f2f278e4640 1 -- 172.21.15.42:0/2945621403 shutdown_connections 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.198+0000 7f2f278e4640 1 -- 172.21.15.42:0/2945621403 wait complete. 2024-03-25T18:23:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.198+0000 7f2f278e4640 10 monclient: build_initial_monmap 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.198+0000 7f2f278e4640 10 monclient: monmap: 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 1 librados: starting msgr at 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 1 librados: starting objecter 2024-03-25T18:23:29.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 1 Processor -- start 2024-03-25T18:23:29.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 1 -- start start 2024-03-25T18:23:29.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 1 librados: setting wanted keys 2024-03-25T18:23:29.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 1 librados: calling monclient init 2024-03-25T18:23:29.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 10 monclient: init 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.199+0000 7f2f278e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f200a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f2013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f201430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f201db6a0 con 0x7f2f200a98f0 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f200abd30 con 0x7f2f201430c0 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20137f30 con 0x7f2f2013f710 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:29.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.200+0000 7f2f278e4640 10 monclient(hunting): authenticate will time out at 7524.425657s 2024-03-25T18:23:29.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.213+0000 7f2f25e81640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2f201430c0 0x7f2f201e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34128/0 (socket says 172.21.15.42:34128) 2024-03-25T18:23:29.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.213+0000 7f2f25e81640 1 -- 172.21.15.42:0/3795012005 learned_addr learned my addr 172.21.15.42:0/3795012005 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:29.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.213+0000 7f2f15ffb640 10 client.?.objecter ms_handle_connect 0x7f2f201430c0 2024-03-25T18:23:29.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.213+0000 7f2f15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:29.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.220+0000 7f2f15ffb640 10 client.?.objecter ms_handle_connect 0x7f2f2013f710 2024-03-25T18:23:29.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.220+0000 7f2f15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:29.224 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.224+0000 7f2f15ffb640 10 client.?.objecter ms_handle_connect 0x7f2f200a98f0 2024-03-25T18:23:29.224 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.224+0000 7f2f15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:29.226 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.226+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711027692 0 0) 0x7f2f200abd30 con 0x7f2f201430c0 2024-03-25T18:23:29.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.227+0000 7f2f15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.227+0000 7f2f15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.227+0000 7f2f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.227+0000 7f2f15ffb640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:23:29.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.227+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f04004eb0 con 0x7f2f201430c0 2024-03-25T18:23:29.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.241+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307760432 0 0) 0x7f2f20137f30 con 0x7f2f2013f710 2024-03-25T18:23:29.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.241+0000 7f2f15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.241+0000 7f2f15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.241+0000 7f2f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.241+0000 7f2f15ffb640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:23:29.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.241+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f200abd30 con 0x7f2f2013f710 2024-03-25T18:23:29.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.242+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4122948297 0 0) 0x7f2f04004eb0 con 0x7f2f201430c0 2024-03-25T18:23:29.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.242+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f20137f30 con 0x7f2f201430c0 2024-03-25T18:23:29.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.242+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f18001c10 con 0x7f2f201430c0 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.242+0000 7f2f15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.242+0000 7f2f15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.242+0000 7f2f15ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:29.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.242+0000 7f2f15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:29.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.245+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551700598 0 0) 0x7f2f201db6a0 con 0x7f2f200a98f0 2024-03-25T18:23:29.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.245+0000 7f2f15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.245+0000 7f2f15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.245+0000 7f2f15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.245+0000 7f2f15ffb640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:23:29.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.245+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f04004eb0 con 0x7f2f200a98f0 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1304580632 0 0) 0x7f2f20137f30 con 0x7f2f201430c0 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 >> v1:172.21.15.42:6790/0 conn(0x7f2f2013f710 legacy=0x7f2f20138530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 >> v1:172.21.15.42:6789/0 conn(0x7f2f200a98f0 legacy=0x7f2f201daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 1 monclient: found mon.b 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20138270 con 0x7f2f201430c0 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:29.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.263+0000 7f2f278e4640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 10 monclient: _renew_subs 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f20138c40 con 0x7f2f201430c0 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 10 monclient: _renew_subs 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f20137bd0 con 0x7f2f201430c0 2024-03-25T18:23:29.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.264+0000 7f2f278e4640 1 librados: init done 2024-03-25T18:23:29.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.265+0000 7f2f278e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:29.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.265+0000 7f2f278e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:29.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.265+0000 7f2f278e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:29.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.265+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f201380c0 con 0x7f2f201430c0 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.560+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f18004b60 con 0x7f2f201430c0 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.560+0000 7f2f15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.560+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f18004e70 con 0x7f2f201430c0 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.560+0000 7f2f15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.560+0000 7f2f15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.560+0000 7f2f15ffb640 10 monclient: dump: 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:29.561 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:29.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.560+0000 7f2f177fe640 4 set_mon_vals no callback set 2024-03-25T18:23:29.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.561+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f2f1801ae40 con 0x7f2f201430c0 2024-03-25T18:23:29.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.561+0000 7f2f25680640 10 monclient: get_auth_request con 0x7f2f0406ddb0 auth_method 0 2024-03-25T18:23:29.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.562+0000 7f2f15ffb640 10 client.4399.objecter ms_handle_connect 0x7f2f0406ddb0 2024-03-25T18:23:29.566 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.565+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f2f18054b70 con 0x7f2f201430c0 2024-03-25T18:23:29.566 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.565+0000 7f2f15ffb640 10 client.4399.objecter ms_dispatch 0x7f2f20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:29.566 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.565+0000 7f2f15ffb640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:29.566 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.565+0000 7f2f15ffb640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:29.566 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.565+0000 7f2f15ffb640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T18:23:29.567 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.566+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f180504b0 con 0x7f2f201430c0 2024-03-25T18:23:29.567 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.566+0000 7f2f15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:29.567 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.566+0000 7f2f15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:29.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.823+0000 7f2f278e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:29.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.823+0000 7f2f278e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:29.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.823+0000 7f2f278e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:29.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.823+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f200627b0 con 0x7f2f201430c0 2024-03-25T18:23:29.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.856+0000 7f2f15ffb640 1 -- 172.21.15.42:0/3795012005 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f2f180525b0 con 0x7f2f201430c0 2024-03-25T18:23:29.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.856+0000 7f2f15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:29.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.856+0000 7f2f15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:29.857 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:23:29.857 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.858+0000 7f2f278e4640 10 librados: watch_flush enter 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.858+0000 7f2f278e4640 10 librados: watch_flush exit 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.858+0000 7f2f278e4640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.858+0000 7f2f278e4640 10 client.4399.objecter successfully canceled tick 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 >> v1:172.21.15.42:6800/3081750888 conn(0x7f2f0406ddb0 legacy=0x7f2f04070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 10 monclient: shutdown 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:29.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 >> v1:172.21.15.120:6789/0 conn(0x7f2f201430c0 legacy=0x7f2f201e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 shutdown_connections 2024-03-25T18:23:29.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 >> 172.21.15.42:0/3795012005 conn(0x7f2f200b7ea0 msgr2=0x7f2f20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:29.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 shutdown_connections 2024-03-25T18:23:29.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 1 -- 172.21.15.42:0/3795012005 wait complete. 2024-03-25T18:23:29.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:29.859+0000 7f2f278e4640 1 librados: shutdown 2024-03-25T18:23:29.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:36.679 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:36.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.792+0000 7f49ea686640 10 monclient: get_monmap_and_config 2024-03-25T18:23:36.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.792+0000 7f49ea686640 10 monclient: build_initial_monmap 2024-03-25T18:23:36.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.792+0000 7f49ea686640 10 monclient: monmap: 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:36.793701+0000 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:36.793701+0000 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:36.794 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.793+0000 7f49ea686640 1 Processor -- start 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.793+0000 7f49ea686640 1 -- start start 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.793+0000 7f49ea686640 10 monclient: init 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 10 monclient(hunting): picked mon.noname-c con 0x7f49e40a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 10 monclient(hunting): picked mon.noname-b con 0x7f49e413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 10 monclient(hunting): picked mon.noname-a con 0x7f49e41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e4142520 con 0x7f49e41430c0 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40b28a0 con 0x7f49e40a98f0 2024-03-25T18:23:36.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40b2a30 con 0x7f49e413f710 2024-03-25T18:23:36.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:36.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.794+0000 7f49ea686640 10 monclient(hunting): authenticate will time out at 7532.020805s 2024-03-25T18:23:36.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.809+0000 7f49e37fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f49e413f710 0x7f49e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:41966/0 (socket says 172.21.15.42:41966) 2024-03-25T18:23:36.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.809+0000 7f49e37fe640 1 -- 172.21.15.42:0/2574885178 learned_addr learned my addr 172.21.15.42:0/2574885178 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:36.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.818+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579246127 0 0) 0x7f49e40b2a30 con 0x7f49e413f710 2024-03-25T18:23:36.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.819+0000 7f49e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.819+0000 7f49e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.819+0000 7f49e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.819+0000 7f49e2ffd640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:23:36.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.819+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc003520 con 0x7f49e413f710 2024-03-25T18:23:36.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.831+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969272759 0 0) 0x7f49e40b28a0 con 0x7f49e40a98f0 2024-03-25T18:23:36.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.831+0000 7f49e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.831+0000 7f49e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.831+0000 7f49e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.831+0000 7f49e2ffd640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:23:36.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.831+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e40b2a30 con 0x7f49e40a98f0 2024-03-25T18:23:36.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.839+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561674771 0 0) 0x7f49e4142520 con 0x7f49e41430c0 2024-03-25T18:23:36.841 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.839+0000 7f49e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.841 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.839+0000 7f49e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.841 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.839+0000 7f49e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.841 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.839+0000 7f49e2ffd640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:23:36.841 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.839+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e40b28a0 con 0x7f49e41430c0 2024-03-25T18:23:36.845 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.843+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2293568808 0 0) 0x7f49cc003520 con 0x7f49e413f710 2024-03-25T18:23:36.845 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 >> v1:172.21.15.120:6789/0 conn(0x7f49e40a98f0 legacy=0x7f49e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 >> v1:172.21.15.42:6789/0 conn(0x7f49e41430c0 legacy=0x7f49e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e40b44c0 con 0x7f49e413f710 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f49d8001690 con 0x7f49e413f710 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49ea686640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49ea686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49e2ffd640 10 monclient: dump: 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:36.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.844+0000 7f49ea686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49d80017f0 con 0x7f49e413f710 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49e2ffd640 1 -- 172.21.15.42:0/2574885178 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f49d8001c00 con 0x7f49e413f710 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49e2ffd640 10 monclient: dump: 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:36.855 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49ea686640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:36.856 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49ea686640 4 set_mon_vals no callback set 2024-03-25T18:23:36.858 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49ea686640 10 monclient: shutdown 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49ea686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.854+0000 7f49ea686640 1 -- 172.21.15.42:0/2574885178 >> v1:172.21.15.42:6790/0 conn(0x7f49e413f710 legacy=0x7f49e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.855+0000 7f49ea686640 1 -- 172.21.15.42:0/2574885178 shutdown_connections 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.855+0000 7f49ea686640 1 -- 172.21.15.42:0/2574885178 >> 172.21.15.42:0/2574885178 conn(0x7f49e40b7ea0 msgr2=0x7f49e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.855+0000 7f49ea686640 1 -- 172.21.15.42:0/2574885178 shutdown_connections 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.855+0000 7f49ea686640 1 -- 172.21.15.42:0/2574885178 wait complete. 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.855+0000 7f49ea686640 10 monclient: build_initial_monmap 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.855+0000 7f49ea686640 10 monclient: monmap: 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:36.859 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 1 librados: starting msgr at 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 1 librados: starting objecter 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 1 Processor -- start 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 1 -- start start 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 1 librados: setting wanted keys 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 1 librados: calling monclient init 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 10 monclient: init 2024-03-25T18:23:36.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 10 monclient(hunting): picked mon.noname-b con 0x7f49e40a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 10 monclient(hunting): picked mon.noname-c con 0x7f49e413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.856+0000 7f49ea686640 10 monclient(hunting): picked mon.noname-a con 0x7f49e41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.857+0000 7f49ea686640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40abc80 con 0x7f49e41430c0 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.857+0000 7f49ea686640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e41d7780 con 0x7f49e40a98f0 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.857+0000 7f49ea686640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e41399a0 con 0x7f49e413f710 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.857+0000 7f49ea686640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.857+0000 7f49ea686640 10 monclient(hunting): authenticate will time out at 7532.083590s 2024-03-25T18:23:36.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.860+0000 7f49e8c23640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f49e41430c0 0x7f49e4138600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41894/0 (socket says 172.21.15.42:41894) 2024-03-25T18:23:36.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.860+0000 7f49e8c23640 1 -- 172.21.15.42:0/4207845984 learned_addr learned my addr 172.21.15.42:0/4207845984 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:36.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.864+0000 7f49e0ff9640 10 client.?.objecter ms_handle_connect 0x7f49e41430c0 2024-03-25T18:23:36.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.864+0000 7f49e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:36.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.864+0000 7f49e0ff9640 10 client.?.objecter ms_handle_connect 0x7f49e413f710 2024-03-25T18:23:36.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.864+0000 7f49e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:36.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.867+0000 7f49e0ff9640 10 client.?.objecter ms_handle_connect 0x7f49e40a98f0 2024-03-25T18:23:36.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.867+0000 7f49e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:36.873 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.872+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253163816 0 0) 0x7f49e40abc80 con 0x7f49e41430c0 2024-03-25T18:23:36.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.872+0000 7f49e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.872+0000 7f49e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.872+0000 7f49e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.872+0000 7f49e0ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:23:36.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.872+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d0004eb0 con 0x7f49e41430c0 2024-03-25T18:23:36.878 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.877+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184526180 0 0) 0x7f49e41399a0 con 0x7f49e413f710 2024-03-25T18:23:36.878 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.877+0000 7f49e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.878 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.877+0000 7f49e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.878 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.877+0000 7f49e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.878 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.877+0000 7f49e0ff9640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:23:36.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.877+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e40abc80 con 0x7f49e413f710 2024-03-25T18:23:36.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.880+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462429337 0 0) 0x7f49d0004eb0 con 0x7f49e41430c0 2024-03-25T18:23:36.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.880+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e41399a0 con 0x7f49e41430c0 2024-03-25T18:23:36.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.880+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f49dc001c70 con 0x7f49e41430c0 2024-03-25T18:23:36.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.880+0000 7f49e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.880+0000 7f49e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.880+0000 7f49e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:36.882 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:36.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.880+0000 7f49e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:36.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.881+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035745404 0 0) 0x7f49e41d7780 con 0x7f49e40a98f0 2024-03-25T18:23:36.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.881+0000 7f49e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.881+0000 7f49e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.881+0000 7f49e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.881+0000 7f49e0ff9640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:23:36.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.881+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d0004eb0 con 0x7f49e40a98f0 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 340195272 0 0) 0x7f49e41399a0 con 0x7f49e41430c0 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 >> v1:172.21.15.42:6790/0 conn(0x7f49e413f710 legacy=0x7f49e41d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 >> v1:172.21.15.120:6789/0 conn(0x7f49e40a98f0 legacy=0x7f49e41d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 1 monclient: found mon.a 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e41d41e0 con 0x7f49e41430c0 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:36.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 10 monclient: _renew_subs 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e41d8370 con 0x7f49e41430c0 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 10 monclient: _renew_subs 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e41d5130 con 0x7f49e41430c0 2024-03-25T18:23:36.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.886+0000 7f49ea686640 1 librados: init done 2024-03-25T18:23:36.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.887+0000 7f49ea686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:36.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.887+0000 7f49ea686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:36.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.887+0000 7f49ea686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:36.890 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:36.887+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e40627b0 con 0x7f49e41430c0 2024-03-25T18:23:37.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.199+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49dc003990 con 0x7f49e41430c0 2024-03-25T18:23:37.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.199+0000 7f49e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:37.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.199+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f49dc003da0 con 0x7f49e41430c0 2024-03-25T18:23:37.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.199+0000 7f49e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:37.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.199+0000 7f49e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:37.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.199+0000 7f49e0ff9640 10 monclient: dump: 2024-03-25T18:23:37.201 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:37.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.199+0000 7f49e27fc640 4 set_mon_vals no callback set 2024-03-25T18:23:37.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.200+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f49dc003fc0 con 0x7f49e41430c0 2024-03-25T18:23:37.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.201+0000 7f49e3fff640 10 monclient: get_auth_request con 0x7f49d006de00 auth_method 0 2024-03-25T18:23:37.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.201+0000 7f49e0ff9640 10 client.4404.objecter ms_handle_connect 0x7f49d006de00 2024-03-25T18:23:37.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f49dc054ae0 con 0x7f49e41430c0 2024-03-25T18:23:37.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 10 client.4404.objecter ms_dispatch 0x7f49e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:37.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:37.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:37.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:23:37.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49dc0505f0 con 0x7f49e41430c0 2024-03-25T18:23:37.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:37.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.210+0000 7f49e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:37.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.427+0000 7f49ea686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:37.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.427+0000 7f49ea686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:37.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.427+0000 7f49ea686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:23:37.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.427+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49e41d4370 con 0x7f49e41430c0 2024-03-25T18:23:37.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.449+0000 7f49e0ff9640 1 -- 172.21.15.42:0/4207845984 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f49dc052520 con 0x7f49e41430c0 2024-03-25T18:23:37.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.449+0000 7f49e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:37.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.449+0000 7f49e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:37.450 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:23:37.451 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 10 librados: watch_flush enter 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 10 librados: watch_flush exit 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 >> v1:172.21.15.42:6800/3081750888 conn(0x7f49d006de00 legacy=0x7f49d00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 10 monclient: shutdown 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 >> v1:172.21.15.42:6789/0 conn(0x7f49e41430c0 legacy=0x7f49e4138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 shutdown_connections 2024-03-25T18:23:37.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.451+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 >> 172.21.15.42:0/4207845984 conn(0x7f49e40b7ea0 msgr2=0x7f49e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:37.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.452+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 shutdown_connections 2024-03-25T18:23:37.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.452+0000 7f49ea686640 1 -- 172.21.15.42:0/4207845984 wait complete. 2024-03-25T18:23:37.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:37.452+0000 7f49ea686640 1 librados: shutdown 2024-03-25T18:23:37.465 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:44.272 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.390+0000 7fd16ca74640 10 monclient: get_monmap_and_config 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.390+0000 7fd16ca74640 10 monclient: build_initial_monmap 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.390+0000 7fd16ca74640 10 monclient: monmap: 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:44.391360+0000 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:44.391360+0000 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:44.392 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:44.392 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:44.392 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:44.392 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:44.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.391+0000 7fd16ca74640 1 Processor -- start 2024-03-25T18:23:44.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.391+0000 7fd16ca74640 1 -- start start 2024-03-25T18:23:44.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.391+0000 7fd16ca74640 10 monclient: init 2024-03-25T18:23:44.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:44.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:44.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 10 monclient(hunting): picked mon.noname-c con 0x7fd1680a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:44.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 10 monclient(hunting): picked mon.noname-b con 0x7fd16813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:44.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 10 monclient(hunting): picked mon.noname-a con 0x7fd1681430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:44.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd168142520 con 0x7fd1681430c0 2024-03-25T18:23:44.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1680b28a0 con 0x7fd1680a98f0 2024-03-25T18:23:44.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1680b2a30 con 0x7fd16813f710 2024-03-25T18:23:44.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:44.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.392+0000 7fd16ca74640 10 monclient(hunting): authenticate will time out at 7539.619023s 2024-03-25T18:23:44.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.396+0000 7fd16659c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd1680a98f0 0x7fd1680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:45730/0 (socket says 172.21.15.42:45730) 2024-03-25T18:23:44.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.396+0000 7fd16659c640 1 -- 172.21.15.42:0/1960789477 learned_addr learned my addr 172.21.15.42:0/1960789477 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:44.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.407+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792982291 0 0) 0x7fd1680b28a0 con 0x7fd1680a98f0 2024-03-25T18:23:44.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.407+0000 7fd16559a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.407+0000 7fd16559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.407+0000 7fd16559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.407+0000 7fd16559a640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:23:44.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.407+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd154003520 con 0x7fd1680a98f0 2024-03-25T18:23:44.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639926140 0 0) 0x7fd154003520 con 0x7fd1680a98f0 2024-03-25T18:23:44.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 >> v1:172.21.15.42:6790/0 conn(0x7fd16813f710 legacy=0x7fd1681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:44.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 >> v1:172.21.15.42:6789/0 conn(0x7fd1681430c0 legacy=0x7fd16814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 10 monclient: _finish_hunting 0 2024-03-25T18:23:44.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 1 monclient: found mon.noname-c 2024-03-25T18:23:44.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1680b44c0 con 0x7fd1680a98f0 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 10 monclient: _finish_auth 0 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 10 monclient: _check_auth_tickets 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd150001690 con 0x7fd1680a98f0 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16559a640 10 monclient: dump: 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.419 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:44.420 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:44.420 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:44.420 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:44.420 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:44.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16ca74640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:23:44.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.417+0000 7fd16ca74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:44.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.421+0000 7fd16559a640 1 -- 172.21.15.42:0/1960789477 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd150001860 con 0x7fd1680a98f0 2024-03-25T18:23:44.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.421+0000 7fd16559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:44.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.421+0000 7fd16ca74640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:44.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.421+0000 7fd16ca74640 4 set_mon_vals no callback set 2024-03-25T18:23:44.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.422+0000 7fd16ca74640 10 monclient: shutdown 2024-03-25T18:23:44.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.422+0000 7fd16ca74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:44.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.422+0000 7fd16ca74640 1 -- 172.21.15.42:0/1960789477 >> v1:172.21.15.120:6789/0 conn(0x7fd1680a98f0 legacy=0x7fd1680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.422+0000 7fd16ca74640 1 -- 172.21.15.42:0/1960789477 shutdown_connections 2024-03-25T18:23:44.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.422+0000 7fd16ca74640 1 -- 172.21.15.42:0/1960789477 >> 172.21.15.42:0/1960789477 conn(0x7fd1680b7ea0 msgr2=0x7fd1680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:44.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.422+0000 7fd16ca74640 1 -- 172.21.15.42:0/1960789477 shutdown_connections 2024-03-25T18:23:44.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.422+0000 7fd16ca74640 1 -- 172.21.15.42:0/1960789477 wait complete. 2024-03-25T18:23:44.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.423+0000 7fd16ca74640 10 monclient: build_initial_monmap 2024-03-25T18:23:44.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.423+0000 7fd16ca74640 10 monclient: monmap: 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:44.425 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.424+0000 7fd16ca74640 1 librados: starting msgr at 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.424+0000 7fd16ca74640 1 librados: starting objecter 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.424+0000 7fd16ca74640 1 Processor -- start 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.424+0000 7fd16ca74640 1 -- start start 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.424+0000 7fd16ca74640 1 librados: setting wanted keys 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.424+0000 7fd16ca74640 1 librados: calling monclient init 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.424+0000 7fd16ca74640 10 monclient: init 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 10 monclient(hunting): picked mon.noname-b con 0x7fd1680a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:44.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 10 monclient(hunting): picked mon.noname-c con 0x7fd16813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:44.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 10 monclient(hunting): picked mon.noname-a con 0x7fd1681430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:44.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1680b07c0 con 0x7fd1681430c0 2024-03-25T18:23:44.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1681d78c0 con 0x7fd1680a98f0 2024-03-25T18:23:44.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1681d7590 con 0x7fd16813f710 2024-03-25T18:23:44.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:44.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.425+0000 7fd16ca74640 10 monclient(hunting): authenticate will time out at 7539.652224s 2024-03-25T18:23:44.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.438+0000 7fd165d9b640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fd16813f710 0x7fd1680b0e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:48842/0 (socket says 172.21.15.42:48842) 2024-03-25T18:23:44.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.438+0000 7fd165d9b640 1 -- 172.21.15.42:0/2129409920 learned_addr learned my addr 172.21.15.42:0/2129409920 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:44.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.439+0000 7fd15effd640 10 client.?.objecter ms_handle_connect 0x7fd1680a98f0 2024-03-25T18:23:44.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.439+0000 7fd15effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:44.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.439+0000 7fd15effd640 10 client.?.objecter ms_handle_connect 0x7fd16813f710 2024-03-25T18:23:44.441 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.439+0000 7fd15effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:44.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.444+0000 7fd15effd640 10 client.?.objecter ms_handle_connect 0x7fd1681430c0 2024-03-25T18:23:44.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.444+0000 7fd15effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:44.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301919547 0 0) 0x7fd1681d7590 con 0x7fd16813f710 2024-03-25T18:23:44.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:23:44.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd148004eb0 con 0x7fd16813f710 2024-03-25T18:23:44.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245787098 0 0) 0x7fd1681d78c0 con 0x7fd1680a98f0 2024-03-25T18:23:44.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:23:44.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.450+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1681d7590 con 0x7fd1680a98f0 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.459+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1764662686 0 0) 0x7fd1681d7590 con 0x7fd1680a98f0 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.460+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1681d78c0 con 0x7fd1680a98f0 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.460+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd150001550 con 0x7fd1680a98f0 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.460+0000 7fd15effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.460+0000 7fd15effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.460+0000 7fd15effd640 10 monclient(hunting): dump: 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.461 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.462 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:44.462 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:44.462 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:44.462 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:44.462 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:44.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.460+0000 7fd15effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:44.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.463+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1499480355 0 0) 0x7fd148004eb0 con 0x7fd16813f710 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.463+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1681d7590 con 0x7fd16813f710 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.463+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd158001960 con 0x7fd16813f710 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.463+0000 7fd15effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.463+0000 7fd15effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.463+0000 7fd15effd640 10 monclient(hunting): dump: 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:44.465 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:44.466 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:44.466 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:44.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2390925772 0 0) 0x7fd1681d78c0 con 0x7fd1680a98f0 2024-03-25T18:23:44.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 >> v1:172.21.15.42:6790/0 conn(0x7fd16813f710 legacy=0x7fd1680b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 >> v1:172.21.15.42:6789/0 conn(0x7fd1681430c0 legacy=0x7fd1681d4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 1 monclient: found mon.b 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1681d41e0 con 0x7fd1680a98f0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 10 monclient: _finish_auth 0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd15effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 10 monclient: _renew_subs 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1680b1560 con 0x7fd1680a98f0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 10 monclient: _renew_subs 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16814dfc0 con 0x7fd1680a98f0 2024-03-25T18:23:44.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.469+0000 7fd16ca74640 1 librados: init done 2024-03-25T18:23:44.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.470+0000 7fd16ca74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:44.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.470+0000 7fd16ca74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:44.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.470+0000 7fd16ca74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:44.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.470+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1680b3ef0 con 0x7fd1680a98f0 2024-03-25T18:23:44.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.743+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd150001210 con 0x7fd1680a98f0 2024-03-25T18:23:44.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.743+0000 7fd15effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:44.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.743+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd150004c30 con 0x7fd1680a98f0 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.743+0000 7fd15effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.743+0000 7fd15effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.743+0000 7fd15effd640 10 monclient: dump: 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:44.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.743+0000 7fd164d99640 4 set_mon_vals no callback set 2024-03-25T18:23:44.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.744+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fd15001ac00 con 0x7fd1680a98f0 2024-03-25T18:23:44.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.744+0000 7fd165d9b640 10 monclient: get_auth_request con 0x7fd14806de40 auth_method 0 2024-03-25T18:23:44.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.745+0000 7fd15effd640 10 client.4411.objecter ms_handle_connect 0x7fd14806de40 2024-03-25T18:23:44.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.746+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fd150054830 con 0x7fd1680a98f0 2024-03-25T18:23:44.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.746+0000 7fd15effd640 10 client.4411.objecter ms_dispatch 0x7fd168000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:44.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.746+0000 7fd15effd640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:44.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.746+0000 7fd15effd640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:44.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.747+0000 7fd15effd640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:23:44.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.749+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd150050170 con 0x7fd1680a98f0 2024-03-25T18:23:44.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.749+0000 7fd15effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:44.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:44.749+0000 7fd15effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:45.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.004+0000 7fd16ca74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:45.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.004+0000 7fd16ca74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:45.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.004+0000 7fd16ca74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:45.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.004+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1681d7720 con 0x7fd1680a98f0 2024-03-25T18:23:45.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.098+0000 7fd15effd640 1 -- 172.21.15.42:0/2129409920 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fd1681d7720 con 0x7fd1680a98f0 2024-03-25T18:23:45.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.098+0000 7fd15effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:45.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.098+0000 7fd15effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:45.100 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:23:45.100 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:45.102 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 10 librados: watch_flush enter 2024-03-25T18:23:45.102 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 10 librados: watch_flush exit 2024-03-25T18:23:45.102 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:23:45.102 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 >> v1:172.21.15.42:6800/3081750888 conn(0x7fd14806de40 legacy=0x7fd148070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 10 monclient: shutdown 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 >> v1:172.21.15.120:6789/0 conn(0x7fd1680a98f0 legacy=0x7fd1681d7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 shutdown_connections 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.100+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 >> 172.21.15.42:0/2129409920 conn(0x7fd1680b7ea0 msgr2=0x7fd168074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.101+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 shutdown_connections 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.101+0000 7fd16ca74640 1 -- 172.21.15.42:0/2129409920 wait complete. 2024-03-25T18:23:45.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:45.101+0000 7fd16ca74640 1 librados: shutdown 2024-03-25T18:23:45.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:51.918 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:52.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.034+0000 7f42fa390640 10 monclient: get_monmap_and_config 2024-03-25T18:23:52.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.034+0000 7f42fa390640 10 monclient: build_initial_monmap 2024-03-25T18:23:52.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.034+0000 7f42fa390640 10 monclient: monmap: 2024-03-25T18:23:52.035 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:52.035 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:52.035 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:52.035399+0000 2024-03-25T18:23:52.035 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:52.035399+0000 2024-03-25T18:23:52.036 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:52.036 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:52.036 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:52.036 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:52.036 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:52.036 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.035+0000 7f42fa390640 1 Processor -- start 2024-03-25T18:23:52.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.035+0000 7f42fa390640 1 -- start start 2024-03-25T18:23:52.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.035+0000 7f42fa390640 10 monclient: init 2024-03-25T18:23:52.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.035+0000 7f42fa390640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:52.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.035+0000 7f42fa390640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:52.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.035+0000 7f42fa390640 10 monclient(hunting): picked mon.noname-b con 0x7f42f40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:52.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.036+0000 7f42fa390640 10 monclient(hunting): picked mon.noname-c con 0x7f42f413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:52.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.036+0000 7f42fa390640 10 monclient(hunting): picked mon.noname-a con 0x7f42f41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:52.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.036+0000 7f42fa390640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f4142520 con 0x7f42f41430c0 2024-03-25T18:23:52.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.036+0000 7f42fa390640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f40b28a0 con 0x7f42f413f710 2024-03-25T18:23:52.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.036+0000 7f42fa390640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f40b2a30 con 0x7f42f40a98f0 2024-03-25T18:23:52.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.036+0000 7f42fa390640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:52.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.036+0000 7f42fa390640 10 monclient(hunting): authenticate will time out at 7547.262837s 2024-03-25T18:23:52.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.041+0000 7f42f37fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f42f413f710 0x7f42f41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:54026/0 (socket says 172.21.15.42:54026) 2024-03-25T18:23:52.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.041+0000 7f42f37fe640 1 -- 172.21.15.42:0/3534605012 learned_addr learned my addr 172.21.15.42:0/3534605012 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:52.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.053+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947196257 0 0) 0x7f42f40b28a0 con 0x7f42f413f710 2024-03-25T18:23:52.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.053+0000 7f42f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.053+0000 7f42f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.053+0000 7f42f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.053+0000 7f42f2ffd640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:23:52.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.053+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42dc003520 con 0x7f42f413f710 2024-03-25T18:23:52.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.064+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605707371 0 0) 0x7f42f40b2a30 con 0x7f42f40a98f0 2024-03-25T18:23:52.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.064+0000 7f42f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.064+0000 7f42f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.064+0000 7f42f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.064+0000 7f42f2ffd640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:23:52.065 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.064+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f40b28a0 con 0x7f42f40a98f0 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2597800177 0 0) 0x7f42dc003520 con 0x7f42f413f710 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 >> v1:172.21.15.42:6790/0 conn(0x7f42f40a98f0 legacy=0x7f42f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 >> v1:172.21.15.42:6789/0 conn(0x7f42f41430c0 legacy=0x7f42f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f40b44c0 con 0x7f42f413f710 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f42e8001690 con 0x7f42f413f710 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42f2ffd640 10 monclient: dump: 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.069 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:52.070 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:52.070 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:52.070 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:52.070 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42fa390640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:23:52.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.067+0000 7f42fa390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42e80017f0 con 0x7f42f413f710 2024-03-25T18:23:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42f2ffd640 1 -- 172.21.15.42:0/3534605012 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f42e8001c00 con 0x7f42f413f710 2024-03-25T18:23:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42fa390640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42fa390640 4 set_mon_vals no callback set 2024-03-25T18:23:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.091+0000 7f42f2ffd640 10 monclient: dump: 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:52.093 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.092+0000 7f42fa390640 10 monclient: shutdown 2024-03-25T18:23:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.092+0000 7f42fa390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.092+0000 7f42fa390640 1 -- 172.21.15.42:0/3534605012 >> v1:172.21.15.120:6789/0 conn(0x7f42f413f710 legacy=0x7f42f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.092+0000 7f42fa390640 1 -- 172.21.15.42:0/3534605012 shutdown_connections 2024-03-25T18:23:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.092+0000 7f42fa390640 1 -- 172.21.15.42:0/3534605012 >> 172.21.15.42:0/3534605012 conn(0x7f42f40b7ea0 msgr2=0x7f42f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.092+0000 7f42fa390640 1 -- 172.21.15.42:0/3534605012 shutdown_connections 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.092+0000 7f42fa390640 1 -- 172.21.15.42:0/3534605012 wait complete. 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.093+0000 7f42fa390640 10 monclient: build_initial_monmap 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.093+0000 7f42fa390640 10 monclient: monmap: 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:52.095 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.094+0000 7f42fa390640 1 librados: starting msgr at 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.094+0000 7f42fa390640 1 librados: starting objecter 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.094+0000 7f42fa390640 1 Processor -- start 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.094+0000 7f42fa390640 1 -- start start 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.094+0000 7f42fa390640 1 librados: setting wanted keys 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.094+0000 7f42fa390640 1 librados: calling monclient init 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.094+0000 7f42fa390640 10 monclient: init 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 10 monclient(hunting): picked mon.noname-a con 0x7f42f40a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 10 monclient(hunting): picked mon.noname-b con 0x7f42f413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 10 monclient(hunting): picked mon.noname-c con 0x7f42f41430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f413b8f0 con 0x7f42f40a98f0 2024-03-25T18:23:52.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f413b510 con 0x7f42f413f710 2024-03-25T18:23:52.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f413b6a0 con 0x7f42f41430c0 2024-03-25T18:23:52.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:52.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.095+0000 7f42fa390640 10 monclient(hunting): authenticate will time out at 7547.321851s 2024-03-25T18:23:52.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.115+0000 7f42f37fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f42f413f710 0x7f42f414e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:54078/0 (socket says 172.21.15.42:54078) 2024-03-25T18:23:52.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.115+0000 7f42f37fe640 1 -- 172.21.15.42:0/1267787287 learned_addr learned my addr 172.21.15.42:0/1267787287 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:52.117 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.116+0000 7f42f0ff9640 10 client.?.objecter ms_handle_connect 0x7f42f413f710 2024-03-25T18:23:52.117 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.116+0000 7f42f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:52.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.118+0000 7f42f0ff9640 10 client.?.objecter ms_handle_connect 0x7f42f40a98f0 2024-03-25T18:23:52.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.118+0000 7f42f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:52.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.124+0000 7f42f0ff9640 10 client.?.objecter ms_handle_connect 0x7f42f41430c0 2024-03-25T18:23:52.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.124+0000 7f42f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:52.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.137+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603452474 0 0) 0x7f42f413b510 con 0x7f42f413f710 2024-03-25T18:23:52.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.137+0000 7f42f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.137+0000 7f42f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.137+0000 7f42f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.137+0000 7f42f0ff9640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:23:52.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.137+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e0004eb0 con 0x7f42f413f710 2024-03-25T18:23:52.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.143+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208445125 0 0) 0x7f42f413b8f0 con 0x7f42f40a98f0 2024-03-25T18:23:52.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.143+0000 7f42f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.143+0000 7f42f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.143+0000 7f42f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.143+0000 7f42f0ff9640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:23:52.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.143+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f413b510 con 0x7f42f40a98f0 2024-03-25T18:23:52.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.145+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986608270 0 0) 0x7f42f413b6a0 con 0x7f42f41430c0 2024-03-25T18:23:52.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.145+0000 7f42f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.145+0000 7f42f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.145+0000 7f42f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.145+0000 7f42f0ff9640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:23:52.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.145+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f413b8f0 con 0x7f42f41430c0 2024-03-25T18:23:52.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.158+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899071998 0 0) 0x7f42e0004eb0 con 0x7f42f413f710 2024-03-25T18:23:52.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.158+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42f413b6a0 con 0x7f42f413f710 2024-03-25T18:23:52.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.158+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f42e8001c40 con 0x7f42f413f710 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.158+0000 7f42f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.158+0000 7f42f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.158+0000 7f42f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.158+0000 7f42f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.164+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 969435502 0 0) 0x7f42f413b8f0 con 0x7f42f41430c0 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.164+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e0004eb0 con 0x7f42f41430c0 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.164+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f42ec001a60 con 0x7f42f41430c0 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.164+0000 7f42f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.164+0000 7f42f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.164+0000 7f42f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.173+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 936042700 0 0) 0x7f42f413b510 con 0x7f42f40a98f0 2024-03-25T18:23:52.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.173+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42f413b8f0 con 0x7f42f40a98f0 2024-03-25T18:23:52.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.173+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f42e4001930 con 0x7f42f40a98f0 2024-03-25T18:23:52.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.173+0000 7f42f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.173+0000 7f42f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.173+0000 7f42f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:52.175 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1164893603 0 0) 0x7f42f413b6a0 con 0x7f42f413f710 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 >> v1:172.21.15.42:6790/0 conn(0x7f42f41430c0 legacy=0x7f42f414e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 >> v1:172.21.15.42:6789/0 conn(0x7f42f40a98f0 legacy=0x7f42f4135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 1 monclient: found mon.b 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f414c9a0 con 0x7f42f413f710 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:52.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 10 monclient: _renew_subs 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f414c4e0 con 0x7f42f413f710 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 10 monclient: _renew_subs 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f4135910 con 0x7f42f413f710 2024-03-25T18:23:52.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.175+0000 7f42fa390640 1 librados: init done 2024-03-25T18:23:52.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.176+0000 7f42fa390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:52.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.176+0000 7f42fa390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:52.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.176+0000 7f42fa390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:52.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.176+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f413ba80 con 0x7f42f413f710 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.457+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42e8004a90 con 0x7f42f413f710 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.457+0000 7f42f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.457+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f42e8004de0 con 0x7f42f413f710 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.457+0000 7f42f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.457+0000 7f42f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.457+0000 7f42f0ff9640 10 monclient: dump: 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:52.459 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:52.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.457+0000 7f42f27fc640 4 set_mon_vals no callback set 2024-03-25T18:23:52.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.458+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f42e8005000 con 0x7f42f413f710 2024-03-25T18:23:52.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.459+0000 7f42f3fff640 10 monclient: get_auth_request con 0x7f42e006e0f0 auth_method 0 2024-03-25T18:23:52.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.459+0000 7f42f0ff9640 10 client.4417.objecter ms_handle_connect 0x7f42e006e0f0 2024-03-25T18:23:52.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f42e8054980 con 0x7f42f413f710 2024-03-25T18:23:52.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 10 client.4417.objecter ms_dispatch 0x7f42f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:52.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:52.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:52.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T18:23:52.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42e8050490 con 0x7f42f413f710 2024-03-25T18:23:52.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:52.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.464+0000 7f42f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:52.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.683+0000 7f42fa390640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:52.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.683+0000 7f42fa390640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:52.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.683+0000 7f42fa390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:52.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.683+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42f414dda0 con 0x7f42f413f710 2024-03-25T18:23:52.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.805+0000 7f42f0ff9640 1 -- 172.21.15.42:0/1267787287 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f42e80523c0 con 0x7f42f413f710 2024-03-25T18:23:52.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.805+0000 7f42f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:52.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.805+0000 7f42f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:52.806 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:23:52.807 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:52.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 10 librados: watch_flush enter 2024-03-25T18:23:52.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 10 librados: watch_flush exit 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 10 client.4417.objecter successfully canceled tick 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 >> v1:172.21.15.42:6800/3081750888 conn(0x7f42e006e0f0 legacy=0x7f42e00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 10 monclient: shutdown 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.807+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 >> v1:172.21.15.120:6789/0 conn(0x7f42f413f710 legacy=0x7f42f414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.808+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 shutdown_connections 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.808+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 >> 172.21.15.42:0/1267787287 conn(0x7f42f40b7ea0 msgr2=0x7f42f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.808+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 shutdown_connections 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.808+0000 7f42fa390640 1 -- 172.21.15.42:0/1267787287 wait complete. 2024-03-25T18:23:52.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:52.808+0000 7f42fa390640 1 librados: shutdown 2024-03-25T18:23:52.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:59.627 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.741+0000 7f2f9404a640 10 monclient: get_monmap_and_config 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.741+0000 7f2f9404a640 10 monclient: build_initial_monmap 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.741+0000 7f2f9404a640 10 monclient: monmap: 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:23:59.742096+0000 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:23:59.742096+0000 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:59.742 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:59.743 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:59.743 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:23:59.743 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:23:59.743 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.741+0000 7f2f9404a640 1 Processor -- start 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 1 -- start start 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient: init 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c0b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c14be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0af940 con 0x7f2f8c14be90 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0b0a70 con 0x7f2f8c0b1730 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0b0d50 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:59.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.742+0000 7f2f9404a640 10 monclient(hunting): authenticate will time out at 7554.969452s 2024-03-25T18:23:59.754 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.753+0000 7f2f91de6640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b4d60 0x7f2f8c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:38910/0 (socket says 172.21.15.42:38910) 2024-03-25T18:23:59.754 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.753+0000 7f2f91de6640 1 -- 172.21.15.42:0/2720905340 learned_addr learned my addr 172.21.15.42:0/2720905340 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:59.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.769+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213927167 0 0) 0x7f2f8c0b0d50 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.769+0000 7f2f90de4640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.769+0000 7f2f90de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.769+0000 7f2f90de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.769+0000 7f2f90de4640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:23:59.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.769+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f78003520 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.773+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124288333 0 0) 0x7f2f8c0b0a70 con 0x7f2f8c0b1730 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.773+0000 7f2f90de4640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.773+0000 7f2f90de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.773+0000 7f2f90de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.773+0000 7f2f90de4640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.773+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8c0b0d50 con 0x7f2f8c0b1730 2024-03-25T18:23:59.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.783+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891789870 0 0) 0x7f2f8c0af940 con 0x7f2f8c14be90 2024-03-25T18:23:59.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.783+0000 7f2f90de4640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.783+0000 7f2f90de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.783+0000 7f2f90de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.783+0000 7f2f90de4640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:23:59.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.783+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8c0b0a70 con 0x7f2f8c14be90 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225065399 0 0) 0x7f2f8c0b0d50 con 0x7f2f8c0b1730 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b4d60 legacy=0x7f2f8c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 >> v1:172.21.15.42:6789/0 conn(0x7f2f8c14be90 legacy=0x7f2f8c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 10 monclient: _finish_hunting 0 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 1 monclient: found mon.noname-c 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c0b1030 con 0x7f2f8c0b1730 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 10 monclient: _finish_auth 0 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 10 monclient: _check_auth_tickets 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f7c001670 con 0x7f2f8c0b1730 2024-03-25T18:23:59.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f90de4640 10 monclient: dump: 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f9404a640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:23:59.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.795+0000 7f2f9404a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f7c001890 con 0x7f2f8c0b1730 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f90de4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f90de4640 1 -- 172.21.15.42:0/2720905340 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f7c001c20 con 0x7f2f8c0b1730 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f9404a640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f9404a640 4 set_mon_vals no callback set 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f90de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f90de4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.816+0000 7f2f90de4640 10 monclient: dump: 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:59.817 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.818 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.818 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:59.818 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:59.818 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:59.818 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:59.818 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:59.818 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:59.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.817+0000 7f2f9404a640 10 monclient: shutdown 2024-03-25T18:23:59.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.817+0000 7f2f9404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.817+0000 7f2f9404a640 1 -- 172.21.15.42:0/2720905340 >> v1:172.21.15.120:6789/0 conn(0x7f2f8c0b1730 legacy=0x7f2f8c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.817+0000 7f2f9404a640 1 -- 172.21.15.42:0/2720905340 shutdown_connections 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.817+0000 7f2f9404a640 1 -- 172.21.15.42:0/2720905340 >> 172.21.15.42:0/2720905340 conn(0x7f2f8c0617a0 msgr2=0x7f2f8c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.817+0000 7f2f9404a640 1 -- 172.21.15.42:0/2720905340 shutdown_connections 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.817+0000 7f2f9404a640 1 -- 172.21.15.42:0/2720905340 wait complete. 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 10 monclient: build_initial_monmap 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 10 monclient: monmap: 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:23:59.820 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 1 librados: starting msgr at 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 1 librados: starting objecter 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 1 Processor -- start 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 1 -- start start 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 1 librados: setting wanted keys 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 1 librados: calling monclient init 2024-03-25T18:23:59.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.818+0000 7f2f9404a640 10 monclient: init 2024-03-25T18:23:59.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:59.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:59.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c0b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:23:59.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c0b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:23:59.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c14be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:23:59.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c1440d0 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c143bb0 con 0x7f2f8c14be90 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c13e7d0 con 0x7f2f8c0b1730 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.819+0000 7f2f9404a640 10 monclient(hunting): authenticate will time out at 7555.046299s 2024-03-25T18:23:59.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.833+0000 7f2f91de6640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b1730 0x7f2f8c12fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:38980/0 (socket says 172.21.15.42:38980) 2024-03-25T18:23:59.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.833+0000 7f2f91de6640 1 -- 172.21.15.42:0/2154358333 learned_addr learned my addr 172.21.15.42:0/2154358333 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:23:59.835 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.835+0000 7f2f827fc640 10 client.?.objecter ms_handle_connect 0x7f2f8c0b1730 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.835+0000 7f2f827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:59.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.841+0000 7f2f827fc640 10 client.?.objecter ms_handle_connect 0x7f2f8c14be90 2024-03-25T18:23:59.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.841+0000 7f2f827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.846+0000 7f2f827fc640 10 client.?.objecter ms_handle_connect 0x7f2f8c0b4d60 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.846+0000 7f2f827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.847+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374703000 0 0) 0x7f2f8c13e7d0 con 0x7f2f8c0b1730 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.847+0000 7f2f827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.847+0000 7f2f827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.847+0000 7f2f827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.847+0000 7f2f827fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:23:59.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.847+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f70004eb0 con 0x7f2f8c0b1730 2024-03-25T18:23:59.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.851+0000 7f2f91de6640 1 -- 172.21.15.42:0/2154358333 >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b1730 legacy=0x7f2f8c12fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:23:59.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.851+0000 7f2f91de6640 1 -- 172.21.15.42:0/2154358333 >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b1730 legacy=0x7f2f8c12fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:23:59.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.851+0000 7f2f91de6640 1 --1- 172.21.15.42:0/2154358333 >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b1730 0x7f2f8c12fe60 :-1 s=OPENED pgs=28698 cs=1 l=1).handle_message read tag failed 2024-03-25T18:23:59.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.851+0000 7f2f91de6640 1 --1- 172.21.15.42:0/2154358333 >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b1730 0x7f2f8c12fe60 :-1 s=OPENED pgs=28698 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:23:59.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.851+0000 7f2f827fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.42:6790/0 2024-03-25T18:23:59.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.858+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129819530 0 0) 0x7f2f8c1440d0 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.859 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.858+0000 7f2f827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.858+0000 7f2f827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.858+0000 7f2f827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.858+0000 7f2f827fc640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:23:59.860 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.858+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8c13e7d0 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.860+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884992635 0 0) 0x7f2f8c143bb0 con 0x7f2f8c14be90 2024-03-25T18:23:59.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.860+0000 7f2f827fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.860+0000 7f2f827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.860+0000 7f2f827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.860+0000 7f2f827fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:23:59.861 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.860+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8c1440d0 con 0x7f2f8c14be90 2024-03-25T18:23:59.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.892+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704956931 0 0) 0x7f2f8c13e7d0 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.892+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f8c143bb0 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.892+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f7c0015b0 con 0x7f2f8c0b4d60 2024-03-25T18:23:59.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.892+0000 7f2f827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:59.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.892+0000 7f2f827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:59.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.892+0000 7f2f827fc640 10 monclient(hunting): dump: 2024-03-25T18:23:59.893 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:59.894 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.892+0000 7f2f827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:59.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.894+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030569033 0 0) 0x7f2f8c1440d0 con 0x7f2f8c14be90 2024-03-25T18:23:59.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.894+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f8c13e7d0 con 0x7f2f8c14be90 2024-03-25T18:23:59.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.894+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f84001bd0 con 0x7f2f8c14be90 2024-03-25T18:23:59.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.894+0000 7f2f827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:59.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.894+0000 7f2f827fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:59.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.894+0000 7f2f827fc640 10 monclient(hunting): dump: 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:23:59.896 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2862436878 0 0) 0x7f2f8c13e7d0 con 0x7f2f8c14be90 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 >> v1:172.21.15.42:6790/0 conn(0x7f2f8c0b1730 legacy=0x7f2f8c12fe60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 >> v1:172.21.15.42:6789/0 conn(0x7f2f8c0b4d60 legacy=0x7f2f8c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 1 monclient: found mon.b 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c145640 con 0x7f2f8c14be90 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:59.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 10 monclient: _renew_subs 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c130570 con 0x7f2f8c14be90 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 10 monclient: _renew_subs 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c143870 con 0x7f2f8c14be90 2024-03-25T18:23:59.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.911+0000 7f2f9404a640 1 librados: init done 2024-03-25T18:23:59.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.913+0000 7f2f9404a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:59.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.913+0000 7f2f9404a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:59.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.913+0000 7f2f9404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:23:59.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:23:59.913+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c143d40 con 0x7f2f8c14be90 2024-03-25T18:24:00.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f84001fe0 con 0x7f2f8c14be90 2024-03-25T18:24:00.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:00.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2f84003b30 con 0x7f2f8c14be90 2024-03-25T18:24:00.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:00.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f827fc640 10 monclient: dump: 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:00.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f83fff640 4 set_mon_vals no callback set 2024-03-25T18:24:00.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.194+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f2f84019b00 con 0x7f2f8c14be90 2024-03-25T18:24:00.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.195+0000 7f2f91de6640 10 monclient: get_auth_request con 0x7f2f7006e010 auth_method 0 2024-03-25T18:24:00.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.195+0000 7f2f827fc640 10 client.4423.objecter ms_handle_connect 0x7f2f7006e010 2024-03-25T18:24:00.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f2f840536e0 con 0x7f2f8c14be90 2024-03-25T18:24:00.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 10 client.4423.objecter ms_dispatch 0x7f2f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:00.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:00.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:00.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T18:24:00.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f8404f1f0 con 0x7f2f8c14be90 2024-03-25T18:24:00.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:00.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.201+0000 7f2f827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:00.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.429+0000 7f2f9404a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:00.430 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.429+0000 7f2f9404a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:00.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.429+0000 7f2f9404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:00.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.429+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f8c145300 con 0x7f2f8c14be90 2024-03-25T18:24:00.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.462+0000 7f2f827fc640 1 -- 172.21.15.42:0/2154358333 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f2f84051120 con 0x7f2f8c14be90 2024-03-25T18:24:00.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.462+0000 7f2f827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:00.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.462+0000 7f2f827fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:00.464 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:00.464 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 10 librados: watch_flush enter 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 10 librados: watch_flush exit 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 10 client.4423.objecter successfully canceled tick 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 >> v1:172.21.15.42:6800/3081750888 conn(0x7f2f7006e010 legacy=0x7f2f700704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 10 monclient: shutdown 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 >> v1:172.21.15.120:6789/0 conn(0x7f2f8c14be90 legacy=0x7f2f8c1e2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 shutdown_connections 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 >> 172.21.15.42:0/2154358333 conn(0x7f2f8c0617a0 msgr2=0x7f2f8c0b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:00.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 shutdown_connections 2024-03-25T18:24:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.464+0000 7f2f9404a640 1 -- 172.21.15.42:0/2154358333 wait complete. 2024-03-25T18:24:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:00.465+0000 7f2f9404a640 1 librados: shutdown 2024-03-25T18:24:00.477 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:06.983 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:04:50.152 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.027 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:04:50.148 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.028 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:04:50.149 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.039 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:04:50.208 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.068 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:04:50.189 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.074 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:04:50.243 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.103 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:04:50.273 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.111 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:04:50.281 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.141 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:04:50.311 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:07.283 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:07.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.394+0000 7f419aa86640 10 monclient: get_monmap_and_config 2024-03-25T18:24:07.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.394+0000 7f419aa86640 10 monclient: build_initial_monmap 2024-03-25T18:24:07.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.394+0000 7f419aa86640 10 monclient: monmap: 2024-03-25T18:24:07.395 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:07.395 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:07.395 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:24:07.395614+0000 2024-03-25T18:24:07.395 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:24:07.395614+0000 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.395+0000 7f419aa86640 1 Processor -- start 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.395+0000 7f419aa86640 1 -- start start 2024-03-25T18:24:07.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.395+0000 7f419aa86640 10 monclient: init 2024-03-25T18:24:07.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 10 monclient(hunting): picked mon.noname-c con 0x7f419406ab80 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 10 monclient(hunting): picked mon.noname-b con 0x7f419406af60 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 10 monclient(hunting): picked mon.noname-a con 0x7f41940b00b0 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41940b0500 con 0x7f41940b00b0 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419413ffa0 con 0x7f419406ab80 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41940af520 con 0x7f419406af60 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:07.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.396+0000 7f419aa86640 10 monclient(hunting): authenticate will time out at 7562.623078s 2024-03-25T18:24:07.411 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.410+0000 7f4198822640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f419406ab80 0x7f41940ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:52470/0 (socket says 172.21.15.42:52470) 2024-03-25T18:24:07.411 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.410+0000 7f4198822640 1 -- 172.21.15.42:0/3753877561 learned_addr learned my addr 172.21.15.42:0/3753877561 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:07.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.422+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593305912 0 0) 0x7f419413ffa0 con 0x7f419406ab80 2024-03-25T18:24:07.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.422+0000 7f418b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.422+0000 7f418b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.422+0000 7f418b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.422+0000 7f418b7fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:24:07.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.422+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f417c003520 con 0x7f419406ab80 2024-03-25T18:24:07.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.423+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370838932 0 0) 0x7f41940af520 con 0x7f419406af60 2024-03-25T18:24:07.424 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.423+0000 7f418b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.423+0000 7f418b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.423+0000 7f418b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.423+0000 7f418b7fe640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:24:07.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.423+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f419413ffa0 con 0x7f419406af60 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372236580 0 0) 0x7f417c003520 con 0x7f419406ab80 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 >> v1:172.21.15.42:6790/0 conn(0x7f419406af60 legacy=0x7f419413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 >> v1:172.21.15.42:6789/0 conn(0x7f41940b00b0 legacy=0x7f419414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4194072890 con 0x7f419406ab80 2024-03-25T18:24:07.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4190001690 con 0x7f419406ab80 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.444+0000 7f418b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.445+0000 7f418b7fe640 10 monclient: dump: 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:07.447 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:07.448 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.445+0000 7f419aa86640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:24:07.448 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.445+0000 7f419aa86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41900017f0 con 0x7f419406ab80 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f418b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f418b7fe640 1 -- 172.21.15.42:0/3753877561 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4190001c00 con 0x7f419406ab80 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f418b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f418b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f418b7fe640 10 monclient: dump: 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.466 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f419aa86640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:07.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.465+0000 7f419aa86640 4 set_mon_vals no callback set 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.466+0000 7f419aa86640 10 monclient: shutdown 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.466+0000 7f419aa86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.466+0000 7f419aa86640 1 -- 172.21.15.42:0/3753877561 >> v1:172.21.15.120:6789/0 conn(0x7f419406ab80 legacy=0x7f41940ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.466+0000 7f419aa86640 1 -- 172.21.15.42:0/3753877561 shutdown_connections 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.466+0000 7f419aa86640 1 -- 172.21.15.42:0/3753877561 >> 172.21.15.42:0/3753877561 conn(0x7f41940b4a80 msgr2=0x7f41940b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.466+0000 7f419aa86640 1 -- 172.21.15.42:0/3753877561 shutdown_connections 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.466+0000 7f419aa86640 1 -- 172.21.15.42:0/3753877561 wait complete. 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 10 monclient: build_initial_monmap 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 10 monclient: monmap: 2024-03-25T18:24:07.469 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 1 librados: starting msgr at 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 1 librados: starting objecter 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 1 Processor -- start 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 1 -- start start 2024-03-25T18:24:07.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 1 librados: setting wanted keys 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 1 librados: calling monclient init 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.467+0000 7f419aa86640 10 monclient: init 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 10 monclient(hunting): picked mon.noname-a con 0x7f419406ab80 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 10 monclient(hunting): picked mon.noname-b con 0x7f41940b00b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 10 monclient(hunting): picked mon.noname-c con 0x7f4194145e80 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41941456f0 con 0x7f419406ab80 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41941daca0 con 0x7f41940b00b0 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41941da930 con 0x7f4194145e80 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:07.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.468+0000 7f419aa86640 10 monclient(hunting): authenticate will time out at 7562.694889s 2024-03-25T18:24:07.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.471+0000 7f418bfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f41940b00b0 0x7f41941d73b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:52522/0 (socket says 172.21.15.42:52522) 2024-03-25T18:24:07.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.471+0000 7f418bfff640 1 -- 172.21.15.42:0/1917373550 learned_addr learned my addr 172.21.15.42:0/1917373550 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:07.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.473+0000 7f41897fa640 10 client.?.objecter ms_handle_connect 0x7f41940b00b0 2024-03-25T18:24:07.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.473+0000 7f41897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:07.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.476+0000 7f41897fa640 10 client.?.objecter ms_handle_connect 0x7f419406ab80 2024-03-25T18:24:07.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.476+0000 7f41897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:07.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.478+0000 7f41897fa640 10 client.?.objecter ms_handle_connect 0x7f4194145e80 2024-03-25T18:24:07.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.478+0000 7f41897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:07.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.486+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878050777 0 0) 0x7f41941daca0 con 0x7f41940b00b0 2024-03-25T18:24:07.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.486+0000 7f41897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.486+0000 7f41897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.488 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.486+0000 7f41897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.488 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.486+0000 7f41897fa640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:24:07.488 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.486+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4180004eb0 con 0x7f41940b00b0 2024-03-25T18:24:07.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.489+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242679143 0 0) 0x7f41941456f0 con 0x7f419406ab80 2024-03-25T18:24:07.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.489+0000 7f41897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.489+0000 7f41897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.489+0000 7f41897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.489+0000 7f41897fa640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:24:07.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.489+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41941daca0 con 0x7f419406ab80 2024-03-25T18:24:07.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.493+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280522621 0 0) 0x7f41941da930 con 0x7f4194145e80 2024-03-25T18:24:07.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.493+0000 7f41897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.493+0000 7f41897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.493+0000 7f41897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.493+0000 7f41897fa640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:24:07.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.493+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41941456f0 con 0x7f4194145e80 2024-03-25T18:24:07.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.506+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245075092 0 0) 0x7f41941daca0 con 0x7f419406ab80 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.506+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41941da930 con 0x7f419406ab80 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.506+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4190001b00 con 0x7f419406ab80 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.506+0000 7f41897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.506+0000 7f41897fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.506+0000 7f41897fa640 10 monclient(hunting): dump: 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:07.508 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.506+0000 7f41897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.508+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434084765 0 0) 0x7f4180004eb0 con 0x7f41940b00b0 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.508+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41941daca0 con 0x7f41940b00b0 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.508+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4184001a10 con 0x7f41940b00b0 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.508+0000 7f41897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.508+0000 7f41897fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.508+0000 7f41897fa640 10 monclient(hunting): dump: 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:07.510 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:07.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3019977513 0 0) 0x7f41941da930 con 0x7f419406ab80 2024-03-25T18:24:07.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 >> v1:172.21.15.42:6790/0 conn(0x7f4194145e80 legacy=0x7f41941d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 >> v1:172.21.15.120:6789/0 conn(0x7f41940b00b0 legacy=0x7f41941d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 10 monclient: _finish_hunting 0 2024-03-25T18:24:07.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 1 monclient: found mon.a 2024-03-25T18:24:07.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:07.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41941d81d0 con 0x7f419406ab80 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 10 monclient: _finish_auth 0 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 10 monclient: _check_auth_tickets 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f41897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 10 monclient: _renew_subs 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4194142d80 con 0x7f419406ab80 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 10 monclient: _renew_subs 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41941e3ce0 con 0x7f419406ab80 2024-03-25T18:24:07.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.518+0000 7f419aa86640 1 librados: init done 2024-03-25T18:24:07.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.519+0000 7f419aa86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:07.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.519+0000 7f419aa86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:07.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.519+0000 7f419aa86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:07.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.519+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41941daac0 con 0x7f419406ab80 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.827+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4190004a90 con 0x7f419406ab80 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.827+0000 7f41897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.827+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4190001cc0 con 0x7f419406ab80 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.827+0000 7f41897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.827+0000 7f41897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.827+0000 7f41897fa640 10 monclient: dump: 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:07.828 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.827+0000 7f418affd640 4 set_mon_vals no callback set 2024-03-25T18:24:07.829 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.828+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f419001ac10 con 0x7f419406ab80 2024-03-25T18:24:07.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.828+0000 7f418bfff640 10 monclient: get_auth_request con 0x7f418006df70 auth_method 0 2024-03-25T18:24:07.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.829+0000 7f41897fa640 10 client.4428.objecter ms_handle_connect 0x7f418006df70 2024-03-25T18:24:07.836 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f41900547c0 con 0x7f419406ab80 2024-03-25T18:24:07.836 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 10 client.4428.objecter ms_dispatch 0x7f4194000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:07.836 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:07.836 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:07.837 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T18:24:07.837 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41900502d0 con 0x7f419406ab80 2024-03-25T18:24:07.837 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:07.837 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:07.835+0000 7f41897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:08.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.083+0000 7f419aa86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:08.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.083+0000 7f419aa86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:08.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.083+0000 7f419aa86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:08.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.083+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41940627b0 con 0x7f419406ab80 2024-03-25T18:24:08.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.157+0000 7f41897fa640 1 -- 172.21.15.42:0/1917373550 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f4190052200 con 0x7f419406ab80 2024-03-25T18:24:08.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.157+0000 7f41897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:08.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.157+0000 7f41897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:08.158 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:08.158 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:08.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 10 librados: watch_flush enter 2024-03-25T18:24:08.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 10 librados: watch_flush exit 2024-03-25T18:24:08.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T18:24:08.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 10 client.4428.objecter successfully canceled tick 2024-03-25T18:24:08.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 >> v1:172.21.15.42:6800/3081750888 conn(0x7f418006df70 legacy=0x7f4180070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.161 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 10 monclient: shutdown 2024-03-25T18:24:08.162 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:08.162 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 >> v1:172.21.15.42:6789/0 conn(0x7f419406ab80 legacy=0x7f41941db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.162 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 shutdown_connections 2024-03-25T18:24:08.162 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 >> 172.21.15.42:0/1917373550 conn(0x7f41940b4a80 msgr2=0x7f419406f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:08.162 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 shutdown_connections 2024-03-25T18:24:08.162 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 1 -- 172.21.15.42:0/1917373550 wait complete. 2024-03-25T18:24:08.162 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:08.160+0000 7f419aa86640 1 librados: shutdown 2024-03-25T18:24:08.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:14.980 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.092+0000 7f87c61a3640 10 monclient: get_monmap_and_config 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.092+0000 7f87c61a3640 10 monclient: build_initial_monmap 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.092+0000 7f87c61a3640 10 monclient: monmap: 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:24:15.093400+0000 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:24:15.093400+0000 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:15.093 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:15.094 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:24:15.094 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:24:15.094 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:15.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.093+0000 7f87c61a3640 1 Processor -- start 2024-03-25T18:24:15.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.093+0000 7f87c61a3640 1 -- start start 2024-03-25T18:24:15.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.093+0000 7f87c61a3640 10 monclient: init 2024-03-25T18:24:15.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:15.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:15.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 10 monclient(hunting): picked mon.noname-b con 0x7f87c006ab80 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:15.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 10 monclient(hunting): picked mon.noname-a con 0x7f87c006af60 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:15.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 10 monclient(hunting): picked mon.noname-c con 0x7f87c00b00b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:15.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c00b0500 con 0x7f87c006af60 2024-03-25T18:24:15.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c013ffa0 con 0x7f87c00b00b0 2024-03-25T18:24:15.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c00af520 con 0x7f87c006ab80 2024-03-25T18:24:15.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:15.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.094+0000 7f87c61a3640 10 monclient(hunting): authenticate will time out at 7570.320672s 2024-03-25T18:24:15.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.105+0000 7f87bffff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f87c00b00b0 0x7f87c014dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:51590/0 (socket says 172.21.15.42:51590) 2024-03-25T18:24:15.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.105+0000 7f87bffff640 1 -- 172.21.15.42:0/1207223941 learned_addr learned my addr 172.21.15.42:0/1207223941 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:15.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.124+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337145708 0 0) 0x7f87c013ffa0 con 0x7f87c00b00b0 2024-03-25T18:24:15.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.124+0000 7f87be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.124+0000 7f87be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.124+0000 7f87be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.124+0000 7f87be7fc640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:24:15.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.124+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a8003520 con 0x7f87c00b00b0 2024-03-25T18:24:15.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.126+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243884010 0 0) 0x7f87c00af520 con 0x7f87c006ab80 2024-03-25T18:24:15.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.126+0000 7f87be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.126+0000 7f87be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.126+0000 7f87be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.126+0000 7f87be7fc640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:24:15.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.126+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c013ffa0 con 0x7f87c006ab80 2024-03-25T18:24:15.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.129+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45963375 0 0) 0x7f87c00b0500 con 0x7f87c006af60 2024-03-25T18:24:15.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.129+0000 7f87be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.129+0000 7f87be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.129+0000 7f87be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.129+0000 7f87be7fc640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:24:15.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.129+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c00af520 con 0x7f87c006af60 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3465348453 0 0) 0x7f87a8003520 con 0x7f87c00b00b0 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 >> v1:172.21.15.42:6790/0 conn(0x7f87c006ab80 legacy=0x7f87c00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 >> v1:172.21.15.42:6789/0 conn(0x7f87c006af60 legacy=0x7f87c013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0072890 con 0x7f87c00b00b0 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87c61a3640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f87b8001a10 con 0x7f87c00b00b0 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87be7fc640 10 monclient: dump: 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:15.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.133+0000 7f87c61a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:15.154 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87b8001b70 con 0x7f87c00b00b0 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87be7fc640 1 -- 172.21.15.42:0/1207223941 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f87b8001f80 con 0x7f87c00b00b0 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87be7fc640 10 monclient: dump: 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:15.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87c61a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:15.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.153+0000 7f87c61a3640 4 set_mon_vals no callback set 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.154+0000 7f87c61a3640 10 monclient: shutdown 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.154+0000 7f87c61a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.154+0000 7f87c61a3640 1 -- 172.21.15.42:0/1207223941 >> v1:172.21.15.120:6789/0 conn(0x7f87c00b00b0 legacy=0x7f87c014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.154+0000 7f87c61a3640 1 -- 172.21.15.42:0/1207223941 shutdown_connections 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.154+0000 7f87c61a3640 1 -- 172.21.15.42:0/1207223941 >> 172.21.15.42:0/1207223941 conn(0x7f87c00b4a80 msgr2=0x7f87c00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.154+0000 7f87c61a3640 1 -- 172.21.15.42:0/1207223941 shutdown_connections 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.154+0000 7f87c61a3640 1 -- 172.21.15.42:0/1207223941 wait complete. 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.155+0000 7f87c61a3640 10 monclient: build_initial_monmap 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.155+0000 7f87c61a3640 10 monclient: monmap: 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:24:15.157 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.156+0000 7f87c61a3640 1 librados: starting msgr at 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.156+0000 7f87c61a3640 1 librados: starting objecter 2024-03-25T18:24:15.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.156+0000 7f87c61a3640 1 Processor -- start 2024-03-25T18:24:15.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.156+0000 7f87c61a3640 1 -- start start 2024-03-25T18:24:15.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.156+0000 7f87c61a3640 1 librados: setting wanted keys 2024-03-25T18:24:15.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.156+0000 7f87c61a3640 1 librados: calling monclient init 2024-03-25T18:24:15.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.156+0000 7f87c61a3640 10 monclient: init 2024-03-25T18:24:15.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:15.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:15.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 10 monclient(hunting): picked mon.noname-c con 0x7f87c006ab80 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:15.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 10 monclient(hunting): picked mon.noname-a con 0x7f87c00b00b0 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:15.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 10 monclient(hunting): picked mon.noname-b con 0x7f87c0142f50 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:15.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c01461a0 con 0x7f87c00b00b0 2024-03-25T18:24:15.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c01d7830 con 0x7f87c0142f50 2024-03-25T18:24:15.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c0144500 con 0x7f87c006ab80 2024-03-25T18:24:15.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:15.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.157+0000 7f87c61a3640 10 monclient(hunting): authenticate will time out at 7570.384089s 2024-03-25T18:24:15.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.169+0000 7f87bf7fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f87c006ab80 0x7f87c01da990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:60190/0 (socket says 172.21.15.42:60190) 2024-03-25T18:24:15.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.169+0000 7f87bf7fe640 1 -- 172.21.15.42:0/3509088821 learned_addr learned my addr 172.21.15.42:0/3509088821 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:15.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.173+0000 7f87a7fff640 10 client.?.objecter ms_handle_connect 0x7f87c006ab80 2024-03-25T18:24:15.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.173+0000 7f87a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.176+0000 7f87a7fff640 10 client.?.objecter ms_handle_connect 0x7f87c0142f50 2024-03-25T18:24:15.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.176+0000 7f87a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.184 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.183+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859191623 0 0) 0x7f87c01d7830 con 0x7f87c0142f50 2024-03-25T18:24:15.184 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.183+0000 7f87a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.183+0000 7f87a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.183+0000 7f87a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.183+0000 7f87a7fff640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:24:15.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.183+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ac004eb0 con 0x7f87c0142f50 2024-03-25T18:24:15.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.187+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673932996 0 0) 0x7f87c0144500 con 0x7f87c006ab80 2024-03-25T18:24:15.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.187+0000 7f87a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.187+0000 7f87a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.187+0000 7f87a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.187+0000 7f87a7fff640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:24:15.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.187+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c01d7830 con 0x7f87c006ab80 2024-03-25T18:24:15.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.192+0000 7f87a7fff640 10 client.?.objecter ms_handle_connect 0x7f87c00b00b0 2024-03-25T18:24:15.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.192+0000 7f87a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.197+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593952570 0 0) 0x7f87ac004eb0 con 0x7f87c0142f50 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.197+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87c0144500 con 0x7f87c0142f50 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.197+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f87b8001e80 con 0x7f87c0142f50 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.198+0000 7f87a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.198+0000 7f87a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.198+0000 7f87a7fff640 10 monclient(hunting): dump: 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.199 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.200 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:15.200 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:15.200 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:15.200 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:15.200 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:15.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.198+0000 7f87a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:15.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.200+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906374611 0 0) 0x7f87c01d7830 con 0x7f87c006ab80 2024-03-25T18:24:15.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.200+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87ac004eb0 con 0x7f87c006ab80 2024-03-25T18:24:15.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.200+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f87b00019f0 con 0x7f87c006ab80 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.200+0000 7f87a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.200+0000 7f87a7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.200+0000 7f87a7fff640 10 monclient(hunting): dump: 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:15.202 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:15.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.201+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999756391 0 0) 0x7f87c01461a0 con 0x7f87c00b00b0 2024-03-25T18:24:15.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.201+0000 7f87a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.201+0000 7f87a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.201+0000 7f87a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.201+0000 7f87a7fff640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:24:15.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.201+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c01d7830 con 0x7f87c00b00b0 2024-03-25T18:24:15.204 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3378763013 0 0) 0x7f87c0144500 con 0x7f87c0142f50 2024-03-25T18:24:15.204 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 >> v1:172.21.15.42:6790/0 conn(0x7f87c006ab80 legacy=0x7f87c01da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.204 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 >> v1:172.21.15.42:6789/0 conn(0x7f87c00b00b0 legacy=0x7f87c01d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.204 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 1 monclient: found mon.b 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c01d7450 con 0x7f87c0142f50 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 10 monclient: _renew_subs 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c01d4810 con 0x7f87c0142f50 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 10 monclient: _renew_subs 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:15.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c01d4d90 con 0x7f87c0142f50 2024-03-25T18:24:15.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.203+0000 7f87c61a3640 1 librados: init done 2024-03-25T18:24:15.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.204+0000 7f87c61a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.204+0000 7f87c61a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.204+0000 7f87c61a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:15.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.204+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c01d76a0 con 0x7f87c0142f50 2024-03-25T18:24:15.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.468+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87b8002020 con 0x7f87c0142f50 2024-03-25T18:24:15.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.468+0000 7f87a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:15.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.468+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f87b8004fb0 con 0x7f87c0142f50 2024-03-25T18:24:15.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.468+0000 7f87a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.468+0000 7f87a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:15.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.468+0000 7f87a7fff640 10 monclient: dump: 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:15.470 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.468+0000 7f87bdffb640 4 set_mon_vals no callback set 2024-03-25T18:24:15.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.469+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f87b801af80 con 0x7f87c0142f50 2024-03-25T18:24:15.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.469+0000 7f87bf7fe640 10 monclient: get_auth_request con 0x7f87ac06df50 auth_method 0 2024-03-25T18:24:15.471 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.469+0000 7f87a7fff640 10 client.4435.objecter ms_handle_connect 0x7f87ac06df50 2024-03-25T18:24:15.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f87b8054a70 con 0x7f87c0142f50 2024-03-25T18:24:15.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 10 client.4435.objecter ms_dispatch 0x7f87c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:15.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:15.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:15.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:24:15.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87b8050580 con 0x7f87c0142f50 2024-03-25T18:24:15.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.474+0000 7f87a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:15.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.688+0000 7f87c61a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:15.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.688+0000 7f87c61a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:15.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.688+0000 7f87c61a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:15.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.688+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87c01d79c0 con 0x7f87c0142f50 2024-03-25T18:24:15.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.714+0000 7f87a7fff640 1 -- 172.21.15.42:0/3509088821 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f87b80524b0 con 0x7f87c0142f50 2024-03-25T18:24:15.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.714+0000 7f87a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:15.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.714+0000 7f87a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:15.715 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:15.715 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 10 librados: watch_flush enter 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 10 librados: watch_flush exit 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 >> v1:172.21.15.42:6800/3081750888 conn(0x7f87ac06df50 legacy=0x7f87ac070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 10 monclient: shutdown 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:15.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 >> v1:172.21.15.120:6789/0 conn(0x7f87c0142f50 legacy=0x7f87c0145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 shutdown_connections 2024-03-25T18:24:15.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.716+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 >> 172.21.15.42:0/3509088821 conn(0x7f87c00b4a80 msgr2=0x7f87c00b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:15.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.717+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 shutdown_connections 2024-03-25T18:24:15.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.717+0000 7f87c61a3640 1 -- 172.21.15.42:0/3509088821 wait complete. 2024-03-25T18:24:15.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:15.717+0000 7f87c61a3640 1 librados: shutdown 2024-03-25T18:24:15.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:22.536 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.653+0000 7f420dc3e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.653+0000 7f420dc3e640 10 monclient: build_initial_monmap 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.653+0000 7f420dc3e640 10 monclient: monmap: 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:24:22.654840+0000 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:24:22.654840+0000 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.654+0000 7f420dc3e640 1 Processor -- start 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.654+0000 7f420dc3e640 1 -- start start 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.654+0000 7f420dc3e640 10 monclient: init 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.655+0000 7f420dc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.655+0000 7f420dc3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.655+0000 7f420dc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f42080a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.655+0000 7f420dc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f420813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:22.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.656+0000 7f420dc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f42081430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:22.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.656+0000 7f420dc3e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4208142520 con 0x7f42080a98f0 2024-03-25T18:24:22.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.656+0000 7f420dc3e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42080b28a0 con 0x7f42081430c0 2024-03-25T18:24:22.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.656+0000 7f420dc3e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42080b2a30 con 0x7f420813f710 2024-03-25T18:24:22.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.656+0000 7f420dc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:22.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.656+0000 7f420dc3e640 10 monclient(hunting): authenticate will time out at 7577.882781s 2024-03-25T18:24:22.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.663+0000 7f4207fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f42081430c0 0x7f420814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:35244/0 (socket says 172.21.15.42:35244) 2024-03-25T18:24:22.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.663+0000 7f4207fff640 1 -- 172.21.15.42:0/3158208464 learned_addr learned my addr 172.21.15.42:0/3158208464 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:22.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.681+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439002381 0 0) 0x7f42080b28a0 con 0x7f42081430c0 2024-03-25T18:24:22.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.681+0000 7f42067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.681+0000 7f42067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.681+0000 7f42067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.681+0000 7f42067fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:24:22.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.681+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f0003520 con 0x7f42081430c0 2024-03-25T18:24:22.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.692+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114130380 0 0) 0x7f4208142520 con 0x7f42080a98f0 2024-03-25T18:24:22.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.692+0000 7f42067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.692+0000 7f42067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.692+0000 7f42067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.692+0000 7f42067fc640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:24:22.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.692+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42080b28a0 con 0x7f42080a98f0 2024-03-25T18:24:22.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.695+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031250157 0 0) 0x7f42080b2a30 con 0x7f420813f710 2024-03-25T18:24:22.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.695+0000 7f42067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.695+0000 7f42067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.695+0000 7f42067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.695+0000 7f42067fc640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:24:22.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.695+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208142520 con 0x7f420813f710 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620880193 0 0) 0x7f41f0003520 con 0x7f42081430c0 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 >> v1:172.21.15.42:6790/0 conn(0x7f420813f710 legacy=0x7f42081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 >> v1:172.21.15.42:6789/0 conn(0x7f42080a98f0 legacy=0x7f42080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42080b44c0 con 0x7f42081430c0 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4200001a10 con 0x7f42081430c0 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:22.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f42067fc640 10 monclient: dump: 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f420dc3e640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:24:22.700 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.697+0000 7f420dc3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4200001bf0 con 0x7f42081430c0 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f42067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f42067fc640 1 -- 172.21.15.42:0/3158208464 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4200002000 con 0x7f42081430c0 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f420dc3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f420dc3e640 4 set_mon_vals no callback set 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f42067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f42067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.713+0000 7f42067fc640 10 monclient: dump: 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:22.715 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 10 monclient: shutdown 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 1 -- 172.21.15.42:0/3158208464 >> v1:172.21.15.120:6789/0 conn(0x7f42081430c0 legacy=0x7f420814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 1 -- 172.21.15.42:0/3158208464 shutdown_connections 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 1 -- 172.21.15.42:0/3158208464 >> 172.21.15.42:0/3158208464 conn(0x7f42080b7ea0 msgr2=0x7f42080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:22.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 1 -- 172.21.15.42:0/3158208464 shutdown_connections 2024-03-25T18:24:22.717 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 1 -- 172.21.15.42:0/3158208464 wait complete. 2024-03-25T18:24:22.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 10 monclient: build_initial_monmap 2024-03-25T18:24:22.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.714+0000 7f420dc3e640 10 monclient: monmap: 2024-03-25T18:24:22.718 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:22.718 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:22.718 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.715+0000 7f420dc3e640 1 librados: starting msgr at 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.715+0000 7f420dc3e640 1 librados: starting objecter 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.715+0000 7f420dc3e640 1 Processor -- start 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.715+0000 7f420dc3e640 1 -- start start 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.715+0000 7f420dc3e640 1 librados: setting wanted keys 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.715+0000 7f420dc3e640 1 librados: calling monclient init 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.715+0000 7f420dc3e640 10 monclient: init 2024-03-25T18:24:22.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f42080a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f420813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f42081430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42081d83a0 con 0x7f42081430c0 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42080abe50 con 0x7f42080a98f0 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4208138f30 con 0x7f420813f710 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:22.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.716+0000 7f420dc3e640 10 monclient(hunting): authenticate will time out at 7577.942743s 2024-03-25T18:24:22.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.729+0000 7f4206ffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f420813f710 0x7f4208138820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:55770/0 (socket says 172.21.15.42:55770) 2024-03-25T18:24:22.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.729+0000 7f4206ffd640 1 -- 172.21.15.42:0/2098906950 learned_addr learned my addr 172.21.15.42:0/2098906950 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:22.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.730+0000 7f41effff640 10 client.?.objecter ms_handle_connect 0x7f420813f710 2024-03-25T18:24:22.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.730+0000 7f41effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:22.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.737+0000 7f41effff640 10 client.?.objecter ms_handle_connect 0x7f42080a98f0 2024-03-25T18:24:22.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.737+0000 7f41effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:22.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.743+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269501032 0 0) 0x7f4208138f30 con 0x7f420813f710 2024-03-25T18:24:22.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.743+0000 7f41effff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.743+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.743+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.743+0000 7f41effff640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:24:22.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.743+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f4004eb0 con 0x7f420813f710 2024-03-25T18:24:22.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 10 client.?.objecter ms_handle_connect 0x7f42081430c0 2024-03-25T18:24:22.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:22.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298417036 0 0) 0x7f42080abe50 con 0x7f42080a98f0 2024-03-25T18:24:22.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:24:22.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.744+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208138f30 con 0x7f42080a98f0 2024-03-25T18:24:22.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.751+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3865693007 0 0) 0x7f4208138f30 con 0x7f42080a98f0 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.751+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42080abe50 con 0x7f42080a98f0 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.751+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f41f8001930 con 0x7f42080a98f0 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.751+0000 7f41effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.751+0000 7f41effff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.751+0000 7f41effff640 10 monclient(hunting): dump: 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:22.753 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:22.754 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.751+0000 7f41effff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698521768 0 0) 0x7f42081d83a0 con 0x7f42081430c0 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208138f30 con 0x7f42081430c0 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 510219095 0 0) 0x7f42080abe50 con 0x7f42080a98f0 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 >> v1:172.21.15.42:6790/0 conn(0x7f420813f710 legacy=0x7f4208138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 >> v1:172.21.15.42:6789/0 conn(0x7f42081430c0 legacy=0x7f4208133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 1 monclient: found mon.b 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42081d7490 con 0x7f42080a98f0 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient: _finish_auth 0 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f41effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 10 monclient: _renew_subs 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42081d7770 con 0x7f42080a98f0 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 10 monclient: _renew_subs 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42081381f0 con 0x7f42080a98f0 2024-03-25T18:24:22.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.757+0000 7f420dc3e640 1 librados: init done 2024-03-25T18:24:22.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.759+0000 7f420dc3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:22.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.759+0000 7f420dc3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:22.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.759+0000 7f420dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:22.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:22.759+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42080627b0 con 0x7f42080a98f0 2024-03-25T18:24:23.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.026+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41f8001d40 con 0x7f42080a98f0 2024-03-25T18:24:23.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.026+0000 7f41effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:23.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.026+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f41f80037b0 con 0x7f42080a98f0 2024-03-25T18:24:23.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.026+0000 7f41effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.026+0000 7f41effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:23.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.026+0000 7f41effff640 10 monclient: dump: 2024-03-25T18:24:23.028 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.026+0000 7f42057fa640 4 set_mon_vals no callback set 2024-03-25T18:24:23.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.027+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f41f8019840 con 0x7f42080a98f0 2024-03-25T18:24:23.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.028+0000 7f4206ffd640 10 monclient: get_auth_request con 0x7f41f406de20 auth_method 0 2024-03-25T18:24:23.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.028+0000 7f41effff640 10 client.4441.objecter ms_handle_connect 0x7f41f406de20 2024-03-25T18:24:23.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.031+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f41f80534f0 con 0x7f42080a98f0 2024-03-25T18:24:23.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.031+0000 7f41effff640 10 client.4441.objecter ms_dispatch 0x7f4208000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:23.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.031+0000 7f41effff640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:23.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.031+0000 7f41effff640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:23.033 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.031+0000 7f41effff640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T18:24:23.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.032+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41f804ee30 con 0x7f42080a98f0 2024-03-25T18:24:23.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.032+0000 7f41effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:23.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.032+0000 7f41effff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:23.256 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.255+0000 7f420dc3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:23.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.255+0000 7f420dc3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:23.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.255+0000 7f420dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:23.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.255+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4208069f40 con 0x7f42080a98f0 2024-03-25T18:24:23.293 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.292+0000 7f41effff640 1 -- 172.21.15.42:0/2098906950 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f41f8050f30 con 0x7f42080a98f0 2024-03-25T18:24:23.293 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.292+0000 7f41effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:23.294 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.292+0000 7f41effff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:23.294 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:23.294 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:23.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 10 librados: watch_flush enter 2024-03-25T18:24:23.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 10 librados: watch_flush exit 2024-03-25T18:24:23.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T18:24:23.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 10 client.4441.objecter successfully canceled tick 2024-03-25T18:24:23.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 >> v1:172.21.15.42:6800/3081750888 conn(0x7f41f406de20 legacy=0x7f41f40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 10 monclient: shutdown 2024-03-25T18:24:23.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:23.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 >> v1:172.21.15.120:6789/0 conn(0x7f42080a98f0 legacy=0x7f42081d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 shutdown_connections 2024-03-25T18:24:23.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 >> 172.21.15.42:0/2098906950 conn(0x7f42080b7ea0 msgr2=0x7f4208074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:23.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.295+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 shutdown_connections 2024-03-25T18:24:23.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.296+0000 7f420dc3e640 1 -- 172.21.15.42:0/2098906950 wait complete. 2024-03-25T18:24:23.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:23.296+0000 7f420dc3e640 1 librados: shutdown 2024-03-25T18:24:23.309 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:27.073 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:10.242 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.077 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:10.247 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.111 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:10.280 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.127 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:10.248 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.135 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:10.304 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.140 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:10.260 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.147 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:10.317 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.166 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:10.287 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:27.184 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:10.353 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:30.116 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.236+0000 7fe65c893640 10 monclient: get_monmap_and_config 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.236+0000 7fe65c893640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.236+0000 7fe65c893640 10 monclient: monmap: 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:24:30.237162+0000 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:24:30.237162+0000 2024-03-25T18:24:30.237 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.238 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:30.238 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:30.238 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:24:30.238 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:24:30.238 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.238 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.236+0000 7fe65c893640 1 Processor -- start 2024-03-25T18:24:30.238 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 1 -- start start 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient: init 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient(hunting): picked mon.noname-c con 0x7fe6580a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient(hunting): picked mon.noname-b con 0x7fe65813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient(hunting): picked mon.noname-a con 0x7fe6581430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe658142520 con 0x7fe6581430c0 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6580b28a0 con 0x7fe6580a98f0 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6580b2a30 con 0x7fe65813f710 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.237+0000 7fe65c893640 10 monclient(hunting): authenticate will time out at 7585.464207s 2024-03-25T18:24:30.246 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.245+0000 7fe65659c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fe6580a98f0 0x7fe6580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:51632/0 (socket says 172.21.15.42:51632) 2024-03-25T18:24:30.246 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.245+0000 7fe65659c640 1 -- 172.21.15.42:0/2339047230 learned_addr learned my addr 172.21.15.42:0/2339047230 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:30.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.258+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777166047 0 0) 0x7fe6580b28a0 con 0x7fe6580a98f0 2024-03-25T18:24:30.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.258+0000 7fe655d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.258+0000 7fe655d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.258+0000 7fe655d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.258+0000 7fe655d9b640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:24:30.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.258+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe644003520 con 0x7fe6580a98f0 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251033197 0 0) 0x7fe658142520 con 0x7fe6581430c0 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6580b28a0 con 0x7fe6581430c0 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873002758 0 0) 0x7fe6580b2a30 con 0x7fe65813f710 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:24:30.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.271+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658142520 con 0x7fe65813f710 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1462520349 0 0) 0x7fe644003520 con 0x7fe6580a98f0 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 >> v1:172.21.15.42:6790/0 conn(0x7fe65813f710 legacy=0x7fe6581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 >> v1:172.21.15.42:6789/0 conn(0x7fe6581430c0 legacy=0x7fe65814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6580b44c0 con 0x7fe6580a98f0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe65c893640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe648001690 con 0x7fe6580a98f0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe655d9b640 10 monclient: dump: 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.295+0000 7fe65c893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6480017f0 con 0x7fe6580a98f0 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe655d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe655d9b640 1 -- 172.21.15.42:0/2339047230 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe648001c00 con 0x7fe6580a98f0 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe65c893640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe65c893640 4 set_mon_vals no callback set 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe655d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe655d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.311+0000 7fe655d9b640 10 monclient: dump: 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.312+0000 7fe65c893640 10 monclient: shutdown 2024-03-25T18:24:30.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.312+0000 7fe65c893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:30.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.312+0000 7fe65c893640 1 -- 172.21.15.42:0/2339047230 >> v1:172.21.15.120:6789/0 conn(0x7fe6580a98f0 legacy=0x7fe6580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.312+0000 7fe65c893640 1 -- 172.21.15.42:0/2339047230 shutdown_connections 2024-03-25T18:24:30.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.312+0000 7fe65c893640 1 -- 172.21.15.42:0/2339047230 >> 172.21.15.42:0/2339047230 conn(0x7fe6580b7ea0 msgr2=0x7fe6580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:30.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.312+0000 7fe65c893640 1 -- 172.21.15.42:0/2339047230 shutdown_connections 2024-03-25T18:24:30.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.312+0000 7fe65c893640 1 -- 172.21.15.42:0/2339047230 wait complete. 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.313+0000 7fe65c893640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.313+0000 7fe65c893640 10 monclient: monmap: 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:30.316 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 1 librados: starting msgr at 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 1 librados: starting objecter 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 1 Processor -- start 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 1 -- start start 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 1 librados: setting wanted keys 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 1 librados: calling monclient init 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 10 monclient: init 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 10 monclient(hunting): picked mon.noname-b con 0x7fe6580a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 10 monclient(hunting): picked mon.noname-c con 0x7fe65813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 10 monclient(hunting): picked mon.noname-a con 0x7fe6581430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.314+0000 7fe65c893640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe658139a40 con 0x7fe6581430c0 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.315+0000 7fe65c893640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe658133f20 con 0x7fe6580a98f0 2024-03-25T18:24:30.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.315+0000 7fe65c893640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6580ab950 con 0x7fe65813f710 2024-03-25T18:24:30.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.315+0000 7fe65c893640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.315+0000 7fe65c893640 10 monclient(hunting): authenticate will time out at 7585.541571s 2024-03-25T18:24:30.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.330+0000 7fe64ffff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fe65813f710 0x7fe658133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:57244/0 (socket says 172.21.15.42:57244) 2024-03-25T18:24:30.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.330+0000 7fe64ffff640 1 -- 172.21.15.42:0/2882651797 learned_addr learned my addr 172.21.15.42:0/2882651797 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:30.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.330+0000 7fe64effd640 10 client.?.objecter ms_handle_connect 0x7fe65813f710 2024-03-25T18:24:30.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.330+0000 7fe64effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.338+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272505160 0 0) 0x7fe6580ab950 con 0x7fe65813f710 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.338+0000 7fe64effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.338+0000 7fe64effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.338+0000 7fe64effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.338+0000 7fe64effd640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.338+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe638004eb0 con 0x7fe65813f710 2024-03-25T18:24:30.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.341+0000 7fe64effd640 10 client.?.objecter ms_handle_connect 0x7fe6580a98f0 2024-03-25T18:24:30.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.341+0000 7fe64effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.344+0000 7fe64effd640 10 client.?.objecter ms_handle_connect 0x7fe6581430c0 2024-03-25T18:24:30.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.344+0000 7fe64effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.347+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349615586 0 0) 0x7fe658133f20 con 0x7fe6580a98f0 2024-03-25T18:24:30.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.347+0000 7fe64effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.347+0000 7fe64effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.347+0000 7fe64effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.347+0000 7fe64effd640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:24:30.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.347+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6580ab950 con 0x7fe6580a98f0 2024-03-25T18:24:30.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.349+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328864438 0 0) 0x7fe658139a40 con 0x7fe6581430c0 2024-03-25T18:24:30.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.349+0000 7fe64effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.349+0000 7fe64effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.349+0000 7fe64effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.349+0000 7fe64effd640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:24:30.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.349+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658133f20 con 0x7fe6581430c0 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.360+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3810245713 0 0) 0x7fe6580ab950 con 0x7fe6580a98f0 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.360+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe658139a40 con 0x7fe6580a98f0 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.360+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe648001b00 con 0x7fe6580a98f0 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.360+0000 7fe64effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.360+0000 7fe64effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.360+0000 7fe64effd640 10 monclient(hunting): dump: 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:30.362 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:30.363 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:30.363 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.360+0000 7fe64effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:30.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.362+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 943399238 0 0) 0x7fe658133f20 con 0x7fe6581430c0 2024-03-25T18:24:30.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.362+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6580ab950 con 0x7fe6581430c0 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.362+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe650001bd0 con 0x7fe6581430c0 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.362+0000 7fe64effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.362+0000 7fe64effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.362+0000 7fe64effd640 10 monclient(hunting): dump: 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:30.364 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.368+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066457839 0 0) 0x7fe638004eb0 con 0x7fe65813f710 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.368+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe658133f20 con 0x7fe65813f710 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.368+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe640001a10 con 0x7fe65813f710 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.368+0000 7fe64effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.368+0000 7fe64effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.368+0000 7fe64effd640 10 monclient(hunting): dump: 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:30.369 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.370 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.370 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.370 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:30.370 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:30.370 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:30.370 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:30.370 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3949845755 0 0) 0x7fe658133f20 con 0x7fe65813f710 2024-03-25T18:24:30.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 >> v1:172.21.15.120:6789/0 conn(0x7fe6580a98f0 legacy=0x7fe658138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 >> v1:172.21.15.42:6789/0 conn(0x7fe6581430c0 legacy=0x7fe658133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 1 monclient: found mon.c 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6580abc30 con 0x7fe65813f710 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe64effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 10 monclient: _renew_subs 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6580abf10 con 0x7fe65813f710 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:30.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 10 monclient: _renew_subs 2024-03-25T18:24:30.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:24:30.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe658139370 con 0x7fe65813f710 2024-03-25T18:24:30.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.380+0000 7fe65c893640 1 librados: init done 2024-03-25T18:24:30.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.381+0000 7fe65c893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.381+0000 7fe65c893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.381+0000 7fe65c893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:24:30.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.381+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6580627b0 con 0x7fe65813f710 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe640001c70 con 0x7fe65813f710 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe64effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fe640003a60 con 0x7fe65813f710 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe64effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe64effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe64effd640 10 monclient: dump: 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:30.708 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:30.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe65559a640 4 set_mon_vals no callback set 2024-03-25T18:24:30.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.707+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fe640003c80 con 0x7fe65813f710 2024-03-25T18:24:30.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.708+0000 7fe65659c640 10 monclient: get_auth_request con 0x7fe63806e160 auth_method 0 2024-03-25T18:24:30.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.708+0000 7fe64effd640 10 client.4442.objecter ms_handle_connect 0x7fe63806e160 2024-03-25T18:24:30.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fe640054800 con 0x7fe65813f710 2024-03-25T18:24:30.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 10 client.4442.objecter ms_dispatch 0x7fe658000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:30.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:30.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:30.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T18:24:30.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe640050310 con 0x7fe65813f710 2024-03-25T18:24:30.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.713+0000 7fe64effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:30.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.940+0000 7fe65c893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:30.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.940+0000 7fe65c893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:30.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.940+0000 7fe65c893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:24:30.942 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.940+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe658138980 con 0x7fe65813f710 2024-03-25T18:24:30.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.988+0000 7fe64effd640 1 -- 172.21.15.42:0/2882651797 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fe658138980 con 0x7fe65813f710 2024-03-25T18:24:30.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.988+0000 7fe64effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:30.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.988+0000 7fe64effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:30.990 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:30.990 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 10 librados: watch_flush enter 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 10 librados: watch_flush exit 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 10 client.4442.objecter successfully canceled tick 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 >> v1:172.21.15.42:6800/3081750888 conn(0x7fe63806e160 legacy=0x7fe638070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 10 monclient: shutdown 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 >> v1:172.21.15.42:6790/0 conn(0x7fe65813f710 legacy=0x7fe658133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 shutdown_connections 2024-03-25T18:24:30.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 >> 172.21.15.42:0/2882651797 conn(0x7fe6580b7ea0 msgr2=0x7fe658074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:30.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 shutdown_connections 2024-03-25T18:24:30.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 1 -- 172.21.15.42:0/2882651797 wait complete. 2024-03-25T18:24:30.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:30.991+0000 7fe65c893640 1 librados: shutdown 2024-03-25T18:24:31.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:37.812 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:37.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.929+0000 7fec4d6f6640 10 monclient: get_monmap_and_config 2024-03-25T18:24:37.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.929+0000 7fec4d6f6640 10 monclient: build_initial_monmap 2024-03-25T18:24:37.930 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.929+0000 7fec4d6f6640 10 monclient: monmap: 2024-03-25T18:24:37.930 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:37.930 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:37.930 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:24:37.930459+0000 2024-03-25T18:24:37.931 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:24:37.930459+0000 2024-03-25T18:24:37.931 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:37.931 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:37.931 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:37.931 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:24:37.931 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:24:37.931 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:37.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.930+0000 7fec4d6f6640 1 Processor -- start 2024-03-25T18:24:37.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.930+0000 7fec4d6f6640 1 -- start start 2024-03-25T18:24:37.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.930+0000 7fec4d6f6640 10 monclient: init 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 10 monclient(hunting): picked mon.noname-b con 0x7fec480a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 10 monclient(hunting): picked mon.noname-a con 0x7fec4813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 10 monclient(hunting): picked mon.noname-c con 0x7fec481430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec48142520 con 0x7fec4813f710 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec480b28a0 con 0x7fec481430c0 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec480b2a30 con 0x7fec480a98f0 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:37.933 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.931+0000 7fec4d6f6640 10 monclient(hunting): authenticate will time out at 7593.157825s 2024-03-25T18:24:37.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.935+0000 7fec46ffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fec480a98f0 0x7fec480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:49230/0 (socket says 172.21.15.42:49230) 2024-03-25T18:24:37.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.935+0000 7fec46ffd640 1 -- 172.21.15.42:0/3469492954 learned_addr learned my addr 172.21.15.42:0/3469492954 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:37.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.952+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844984716 0 0) 0x7fec480b2a30 con 0x7fec480a98f0 2024-03-25T18:24:37.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.952+0000 7fec467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:37.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.952+0000 7fec467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:37.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.952+0000 7fec467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:37.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.952+0000 7fec467fc640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:24:37.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.952+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34003520 con 0x7fec480a98f0 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.963+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234919310 0 0) 0x7fec480b28a0 con 0x7fec481430c0 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.963+0000 7fec467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.963+0000 7fec467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.963+0000 7fec467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.963+0000 7fec467fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.963+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec480b2a30 con 0x7fec481430c0 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831173578 0 0) 0x7fec34003520 con 0x7fec480a98f0 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.120:6789/0 conn(0x7fec481430c0 legacy=0x7fec4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6789/0 conn(0x7fec4813f710 legacy=0x7fec481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 1 monclient: found mon.noname-b 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec480b44c0 con 0x7fec480a98f0 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:37.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fec38001690 con 0x7fec480a98f0 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec467fc640 10 monclient: dump: 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:37.967 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:37.968 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:37.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec4d6f6640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:24:37.968 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.964+0000 7fec4d6f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:37.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec46ffd640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6790/0 conn(0x7fec480a98f0 legacy=0x7fec480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec46ffd640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6790/0 conn(0x7fec480a98f0 legacy=0x7fec480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec46ffd640 1 --1- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6790/0 conn(0x7fec480a98f0 0x7fec480a9cd0 :-1 s=OPENED pgs=33844 cs=1 l=1).handle_message read tag failed 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec46ffd640 1 --1- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6790/0 conn(0x7fec480a98f0 0x7fec480a9cd0 :-1 s=OPENED pgs=33844 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.42:6790/0 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6790/0 conn(0x7fec480a98f0 legacy=0x7fec480a9cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient(hunting): picked mon.c con 0x7fec34003cd0 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient(hunting): picked mon.a con 0x7fec4813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient(hunting): picked mon.b con 0x7fec481430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fec34003520 con 0x7fec4813f710 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fec480b28a0 con 0x7fec481430c0 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fec34004520 con 0x7fec34003cd0 2024-03-25T18:24:37.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:37.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.971+0000 7fec467fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.42:6790/0 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.985+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186418236 0 0) 0x7fec34004520 con 0x7fec34003cd0 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.985+0000 7fec467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.985+0000 7fec467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.985+0000 7fec467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.985+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.42:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fec340110e0 con 0x7fec34003cd0 2024-03-25T18:24:37.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3146968797 0 0) 0x7fec340110e0 con 0x7fec34003cd0 2024-03-25T18:24:37.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.120:6789/0 conn(0x7fec481430c0 legacy=0x7fec3400d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6789/0 conn(0x7fec4813f710 legacy=0x7fec34004ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 1 monclient: found mon.c 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fec340121b0 con 0x7fec34003cd0 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:37.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:37.992+0000 7fec467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fec380048a0 con 0x7fec34003cd0 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec467fc640 10 monclient: dump: 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec467fc640 1 -- 172.21.15.42:0/3469492954 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec38004a70 con 0x7fec34003cd0 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec4d6f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.011+0000 7fec4d6f6640 4 set_mon_vals no callback set 2024-03-25T18:24:38.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.012+0000 7fec4d6f6640 10 monclient: shutdown 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.012+0000 7fec4d6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.012+0000 7fec4d6f6640 1 -- 172.21.15.42:0/3469492954 >> v1:172.21.15.42:6790/0 conn(0x7fec34003cd0 legacy=0x7fec340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.012+0000 7fec4d6f6640 1 -- 172.21.15.42:0/3469492954 shutdown_connections 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.012+0000 7fec4d6f6640 1 -- 172.21.15.42:0/3469492954 >> 172.21.15.42:0/3469492954 conn(0x7fec480b7ea0 msgr2=0x7fec480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.012+0000 7fec4d6f6640 1 -- 172.21.15.42:0/3469492954 shutdown_connections 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.012+0000 7fec4d6f6640 1 -- 172.21.15.42:0/3469492954 wait complete. 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 10 monclient: build_initial_monmap 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 10 monclient: monmap: 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:24:38.016 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 1 librados: starting msgr at 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 1 librados: starting objecter 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 1 Processor -- start 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 1 -- start start 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 1 librados: setting wanted keys 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 1 librados: calling monclient init 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.013+0000 7fec4d6f6640 10 monclient: init 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:38.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 10 monclient(hunting): picked mon.noname-c con 0x7fec34003cd0 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 10 monclient(hunting): picked mon.noname-a con 0x7fec4813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 10 monclient(hunting): picked mon.noname-b con 0x7fec481430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec48139a30 con 0x7fec4813f710 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec481d8160 con 0x7fec481430c0 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec481421e0 con 0x7fec34003cd0 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.014+0000 7fec4d6f6640 10 monclient(hunting): authenticate will time out at 7593.241193s 2024-03-25T18:24:38.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.022+0000 7fec46ffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fec34003cd0 0x7fec481d7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:49322/0 (socket says 172.21.15.42:49322) 2024-03-25T18:24:38.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.022+0000 7fec46ffd640 1 -- 172.21.15.42:0/2825497962 learned_addr learned my addr 172.21.15.42:0/2825497962 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:38.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.023+0000 7fec3f7fe640 10 client.?.objecter ms_handle_connect 0x7fec34003cd0 2024-03-25T18:24:38.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.023+0000 7fec3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.026+0000 7fec3f7fe640 10 client.?.objecter ms_handle_connect 0x7fec481430c0 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.026+0000 7fec3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.027+0000 7fec3f7fe640 10 client.?.objecter ms_handle_connect 0x7fec4813f710 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.027+0000 7fec3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.033+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675429197 0 0) 0x7fec481d8160 con 0x7fec481430c0 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.033+0000 7fec3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.033+0000 7fec3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.033+0000 7fec3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.033+0000 7fec3f7fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.033+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec28004eb0 con 0x7fec481430c0 2024-03-25T18:24:38.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.037+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239565884 0 0) 0x7fec481421e0 con 0x7fec34003cd0 2024-03-25T18:24:38.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.037+0000 7fec3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:38.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.037+0000 7fec3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:38.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.037+0000 7fec3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:38.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.037+0000 7fec3f7fe640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:24:38.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.037+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec481d8160 con 0x7fec34003cd0 2024-03-25T18:24:38.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.038+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703682435 0 0) 0x7fec48139a30 con 0x7fec4813f710 2024-03-25T18:24:38.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.038+0000 7fec3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:38.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.038+0000 7fec3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:38.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.038+0000 7fec3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:38.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.038+0000 7fec3f7fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:24:38.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.038+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec481421e0 con 0x7fec4813f710 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.048+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 931247133 0 0) 0x7fec481421e0 con 0x7fec4813f710 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.048+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec48139a30 con 0x7fec4813f710 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.048+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fec300017b0 con 0x7fec4813f710 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.048+0000 7fec3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.048+0000 7fec3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.048+0000 7fec3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.050 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.051 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:38.051 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:38.051 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:38.051 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:38.051 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:38.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.048+0000 7fec3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.054+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1236066991 0 0) 0x7fec481d8160 con 0x7fec34003cd0 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.054+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec481421e0 con 0x7fec34003cd0 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.054+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fec38001220 con 0x7fec34003cd0 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.054+0000 7fec3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.054+0000 7fec3f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.054+0000 7fec3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.056 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.057 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:38.057 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:38.057 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:38.057 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:38.057 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:38.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.055+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2258966743 0 0) 0x7fec28004eb0 con 0x7fec481430c0 2024-03-25T18:24:38.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.055+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec481d8160 con 0x7fec481430c0 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.055+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fec40001b50 con 0x7fec481430c0 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.055+0000 7fec3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.055+0000 7fec3f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.055+0000 7fec3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:38.058 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:38.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1398633438 0 0) 0x7fec48139a30 con 0x7fec4813f710 2024-03-25T18:24:38.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 >> v1:172.21.15.42:6790/0 conn(0x7fec34003cd0 legacy=0x7fec481d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 >> v1:172.21.15.120:6789/0 conn(0x7fec481430c0 legacy=0x7fec48139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 1 monclient: found mon.a 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec480b77f0 con 0x7fec4813f710 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 10 monclient: _renew_subs 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:38.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec480b7a40 con 0x7fec4813f710 2024-03-25T18:24:38.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:38.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 10 monclient: _renew_subs 2024-03-25T18:24:38.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:38.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec480b60c0 con 0x7fec4813f710 2024-03-25T18:24:38.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.060+0000 7fec4d6f6640 1 librados: init done 2024-03-25T18:24:38.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.061+0000 7fec4d6f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.064 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.061+0000 7fec4d6f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.064 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.061+0000 7fec4d6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:38.064 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.061+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec480b3ef0 con 0x7fec4813f710 2024-03-25T18:24:38.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.374+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec30001cc0 con 0x7fec4813f710 2024-03-25T18:24:38.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.374+0000 7fec3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.374+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fec30003790 con 0x7fec4813f710 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.374+0000 7fec3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.374+0000 7fec3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.374+0000 7fec3f7fe640 10 monclient: dump: 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:38.376 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:38.377 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:38.377 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:38.377 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:38.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.374+0000 7fec45ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:38.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.375+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fec300039b0 con 0x7fec4813f710 2024-03-25T18:24:38.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.376+0000 7fec46ffd640 10 monclient: get_auth_request con 0x7fec2806e0f0 auth_method 0 2024-03-25T18:24:38.378 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.376+0000 7fec3f7fe640 10 client.4452.objecter ms_handle_connect 0x7fec2806e0f0 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fec300534a0 con 0x7fec4813f710 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 10 client.4452.objecter ms_dispatch 0x7fec48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec3004efb0 con 0x7fec4813f710 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.383+0000 7fec3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:38.615 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.614+0000 7fec4d6f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:38.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.614+0000 7fec4d6f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:38.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.614+0000 7fec4d6f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:24:38.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.614+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec4813d010 con 0x7fec4813f710 2024-03-25T18:24:38.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.648+0000 7fec3f7fe640 1 -- 172.21.15.42:0/2825497962 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fec30050ee0 con 0x7fec4813f710 2024-03-25T18:24:38.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.648+0000 7fec3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:38.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.648+0000 7fec3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:38.650 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:38.650 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:38.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.650+0000 7fec4d6f6640 10 librados: watch_flush enter 2024-03-25T18:24:38.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 10 librados: watch_flush exit 2024-03-25T18:24:38.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-25T18:24:38.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 10 client.4452.objecter successfully canceled tick 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 >> v1:172.21.15.42:6800/3081750888 conn(0x7fec2806e0f0 legacy=0x7fec280705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 10 monclient: shutdown 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 >> v1:172.21.15.42:6789/0 conn(0x7fec4813f710 legacy=0x7fec48138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 shutdown_connections 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 >> 172.21.15.42:0/2825497962 conn(0x7fec480b7ea0 msgr2=0x7fec4813d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 shutdown_connections 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 1 -- 172.21.15.42:0/2825497962 wait complete. 2024-03-25T18:24:38.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:38.651+0000 7fec4d6f6640 1 librados: shutdown 2024-03-25T18:24:38.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:45.470 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.587+0000 7f2b9cde8640 10 monclient: get_monmap_and_config 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.588+0000 7f2b9cde8640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.588+0000 7f2b9cde8640 10 monclient: monmap: 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:24:45.589066+0000 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:24:45.589066+0000 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:45.589 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:24:45.590 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:24:45.590 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.588+0000 7f2b9cde8640 1 Processor -- start 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.588+0000 7f2b9cde8640 1 -- start start 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.588+0000 7f2b9cde8640 10 monclient: init 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 10 monclient(hunting): picked mon.noname-b con 0x7f2b980a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 10 monclient(hunting): picked mon.noname-c con 0x7f2b981430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b98142520 con 0x7f2b9813f710 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b980b28a0 con 0x7f2b981430c0 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b980b2a30 con 0x7f2b980a98f0 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.589+0000 7f2b9cde8640 10 monclient(hunting): authenticate will time out at 7600.816286s 2024-03-25T18:24:45.601 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.601+0000 7f2b9659c640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f2b980a98f0 0x7f2b980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:53938/0 (socket says 172.21.15.42:53938) 2024-03-25T18:24:45.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.601+0000 7f2b9659c640 1 -- 172.21.15.42:0/4182575457 learned_addr learned my addr 172.21.15.42:0/4182575457 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:45.615 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.614+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710880792 0 0) 0x7f2b98142520 con 0x7f2b9813f710 2024-03-25T18:24:45.615 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.614+0000 7f2b9559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.615 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.614+0000 7f2b9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.614+0000 7f2b9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.614+0000 7f2b9559a640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:24:45.616 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.614+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b80003520 con 0x7f2b9813f710 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217227845 0 0) 0x7f2b980b28a0 con 0x7f2b981430c0 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b98142520 con 0x7f2b981430c0 2024-03-25T18:24:45.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191722866 0 0) 0x7f2b980b2a30 con 0x7f2b980a98f0 2024-03-25T18:24:45.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:24:45.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.616+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b980b28a0 con 0x7f2b980a98f0 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1861020130 0 0) 0x7f2b80003520 con 0x7f2b9813f710 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 >> v1:172.21.15.42:6790/0 conn(0x7f2b980a98f0 legacy=0x7f2b980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 >> v1:172.21.15.120:6789/0 conn(0x7f2b981430c0 legacy=0x7f2b9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 1 monclient: found mon.noname-a 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980b44c0 con 0x7f2b9813f710 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2b8c001690 con 0x7f2b9813f710 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.625+0000 7f2b9559a640 10 monclient: dump: 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.626+0000 7f2b9cde8640 5 monclient: authenticate success, global_id 4455 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.626+0000 7f2b9cde8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b8c0017f0 con 0x7f2b9813f710 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9559a640 1 -- 172.21.15.42:0/4182575457 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2b8c001c00 con 0x7f2b9813f710 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9559a640 10 monclient: dump: 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.653 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9cde8640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:45.654 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.651+0000 7f2b9cde8640 4 set_mon_vals no callback set 2024-03-25T18:24:45.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.652+0000 7f2b9cde8640 10 monclient: shutdown 2024-03-25T18:24:45.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.652+0000 7f2b9cde8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:45.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.652+0000 7f2b9cde8640 1 -- 172.21.15.42:0/4182575457 >> v1:172.21.15.42:6789/0 conn(0x7f2b9813f710 legacy=0x7f2b981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.653+0000 7f2b9cde8640 1 -- 172.21.15.42:0/4182575457 shutdown_connections 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.653+0000 7f2b9cde8640 1 -- 172.21.15.42:0/4182575457 >> 172.21.15.42:0/4182575457 conn(0x7f2b980b7ea0 msgr2=0x7f2b980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.653+0000 7f2b9cde8640 1 -- 172.21.15.42:0/4182575457 shutdown_connections 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.653+0000 7f2b9cde8640 1 -- 172.21.15.42:0/4182575457 wait complete. 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 10 monclient: monmap: 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:45.656 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 1 librados: starting msgr at 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 1 librados: starting objecter 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 1 Processor -- start 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 1 -- start start 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 1 librados: setting wanted keys 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 1 librados: calling monclient init 2024-03-25T18:24:45.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.654+0000 7f2b9cde8640 10 monclient: init 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 10 monclient(hunting): picked mon.noname-c con 0x7f2b980a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 10 monclient(hunting): picked mon.noname-a con 0x7f2b9813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 10 monclient(hunting): picked mon.noname-b con 0x7f2b981430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b981397c0 con 0x7f2b9813f710 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b981daa20 con 0x7f2b981430c0 2024-03-25T18:24:45.658 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b981d8210 con 0x7f2b980a98f0 2024-03-25T18:24:45.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.655+0000 7f2b9cde8640 10 monclient(hunting): authenticate will time out at 7600.882042s 2024-03-25T18:24:45.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.659+0000 7f2b96d9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2b981430c0 0x7f2b981d7b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:42878/0 (socket says 172.21.15.42:42878) 2024-03-25T18:24:45.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.659+0000 7f2b96d9d640 1 -- 172.21.15.42:0/3654669682 learned_addr learned my addr 172.21.15.42:0/3654669682 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:45.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.659+0000 7f2b86ffd640 10 client.?.objecter ms_handle_connect 0x7f2b981430c0 2024-03-25T18:24:45.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.659+0000 7f2b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.666+0000 7f2b86ffd640 10 client.?.objecter ms_handle_connect 0x7f2b980a98f0 2024-03-25T18:24:45.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.666+0000 7f2b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.674+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806309861 0 0) 0x7f2b981daa20 con 0x7f2b981430c0 2024-03-25T18:24:45.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.674+0000 7f2b86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.674+0000 7f2b86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.674+0000 7f2b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.674+0000 7f2b86ffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:24:45.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.674+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78004eb0 con 0x7f2b981430c0 2024-03-25T18:24:45.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.678+0000 7f2b86ffd640 10 client.?.objecter ms_handle_connect 0x7f2b9813f710 2024-03-25T18:24:45.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.678+0000 7f2b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.690+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613651415 0 0) 0x7f2b981d8210 con 0x7f2b980a98f0 2024-03-25T18:24:45.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.690+0000 7f2b86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.690+0000 7f2b86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.690+0000 7f2b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.690+0000 7f2b86ffd640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:24:45.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.690+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b981daa20 con 0x7f2b980a98f0 2024-03-25T18:24:45.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.695+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813685513 0 0) 0x7f2b981397c0 con 0x7f2b9813f710 2024-03-25T18:24:45.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.695+0000 7f2b86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.695+0000 7f2b86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.695+0000 7f2b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.695+0000 7f2b86ffd640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:24:45.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.695+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b981d8210 con 0x7f2b9813f710 2024-03-25T18:24:45.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.700+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042300558 0 0) 0x7f2b981daa20 con 0x7f2b980a98f0 2024-03-25T18:24:45.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.700+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b981397c0 con 0x7f2b980a98f0 2024-03-25T18:24:45.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.700+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2b88001870 con 0x7f2b980a98f0 2024-03-25T18:24:45.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.700+0000 7f2b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.700+0000 7f2b86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.700+0000 7f2b86ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:45.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.700+0000 7f2b86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.701+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 157500748 0 0) 0x7f2b78004eb0 con 0x7f2b981430c0 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.701+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b981daa20 con 0x7f2b981430c0 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.701+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2b90001c90 con 0x7f2b981430c0 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.701+0000 7f2b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.701+0000 7f2b86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.701+0000 7f2b86ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.703 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.704 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.704 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:45.704 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:45.704 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:45.704 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:45.704 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:45.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1840407870 0 0) 0x7f2b981daa20 con 0x7f2b981430c0 2024-03-25T18:24:45.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 >> v1:172.21.15.42:6790/0 conn(0x7f2b980a98f0 legacy=0x7f2b981daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 >> v1:172.21.15.42:6789/0 conn(0x7f2b9813f710 legacy=0x7f2b981d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 1 monclient: found mon.b 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b981db650 con 0x7f2b981430c0 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 10 monclient: _renew_subs 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b98137720 con 0x7f2b981430c0 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 10 monclient: _renew_subs 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b98137d50 con 0x7f2b981430c0 2024-03-25T18:24:45.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.713+0000 7f2b9cde8640 1 librados: init done 2024-03-25T18:24:45.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.714+0000 7f2b9cde8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.714+0000 7f2b9cde8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.714+0000 7f2b9cde8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:45.716 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.714+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b981db840 con 0x7f2b981430c0 2024-03-25T18:24:45.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.723+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b90003970 con 0x7f2b981430c0 2024-03-25T18:24:45.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.723+0000 7f2b86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.723+0000 7f2b94d99640 4 set_mon_vals no callback set 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.733+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2b90003be0 con 0x7f2b981430c0 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.733+0000 7f2b86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.733+0000 7f2b86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.733+0000 7f2b86ffd640 10 monclient: dump: 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.734 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:45.735 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.735 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:45.735 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:45.735 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:45.735 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:45.735 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:45.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.733+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f2b90003dc0 con 0x7f2b981430c0 2024-03-25T18:24:45.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.734+0000 7f2b9659c640 10 monclient: get_auth_request con 0x7f2b7806dfb0 auth_method 0 2024-03-25T18:24:45.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.734+0000 7f2b86ffd640 10 client.4459.objecter ms_handle_connect 0x7f2b7806dfb0 2024-03-25T18:24:45.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.736+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f2b900549d0 con 0x7f2b981430c0 2024-03-25T18:24:45.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.736+0000 7f2b86ffd640 10 client.4459.objecter ms_dispatch 0x7f2b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:45.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.736+0000 7f2b86ffd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:45.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.736+0000 7f2b86ffd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:45.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:45.736+0000 7f2b86ffd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:24:46.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.007+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b900504e0 con 0x7f2b981430c0 2024-03-25T18:24:46.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.007+0000 7f2b86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.007+0000 7f2b86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:46.233 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.232+0000 7f2b9cde8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.232+0000 7f2b9cde8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.232+0000 7f2b9cde8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:46.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.232+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b98074cc0 con 0x7f2b981430c0 2024-03-25T18:24:46.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.269+0000 7f2b86ffd640 1 -- 172.21.15.42:0/3654669682 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f2b90052410 con 0x7f2b981430c0 2024-03-25T18:24:46.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.269+0000 7f2b86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.269+0000 7f2b86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:46.271 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:46.271 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:46.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 10 librados: watch_flush enter 2024-03-25T18:24:46.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 10 librados: watch_flush exit 2024-03-25T18:24:46.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:24:46.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 >> v1:172.21.15.42:6800/3081750888 conn(0x7f2b7806dfb0 legacy=0x7f2b78070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 10 monclient: shutdown 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.271+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 >> v1:172.21.15.120:6789/0 conn(0x7f2b981430c0 legacy=0x7f2b981d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.272+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 shutdown_connections 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.272+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 >> 172.21.15.42:0/3654669682 conn(0x7f2b980b7ea0 msgr2=0x7f2b9813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.272+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 shutdown_connections 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.272+0000 7f2b9cde8640 1 -- 172.21.15.42:0/3654669682 wait complete. 2024-03-25T18:24:46.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:46.272+0000 7f2b9cde8640 1 librados: shutdown 2024-03-25T18:24:46.285 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:47.159 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:30.328 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.164 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:30.333 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.187 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:30.356 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.210 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:30.331 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.216 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:30.337 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.221 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:30.390 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.224 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:30.393 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.236 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:30.356 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:47.240 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:30.409 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.091 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.207+0000 7f2fbf504640 10 monclient: get_monmap_and_config 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.207+0000 7f2fbf504640 10 monclient: build_initial_monmap 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.207+0000 7f2fbf504640 10 monclient: monmap: 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:24:53.208640+0000 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:24:53.208640+0000 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:24:53.209 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:53.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.208+0000 7f2fbf504640 1 Processor -- start 2024-03-25T18:24:53.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.208+0000 7f2fbf504640 1 -- start start 2024-03-25T18:24:53.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.208+0000 7f2fbf504640 10 monclient: init 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb80a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb81430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb8142520 con 0x7f2fb81430c0 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb80b28a0 con 0x7f2fb813f710 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb80b2a30 con 0x7f2fb80a98f0 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:53.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.209+0000 7f2fbf504640 10 monclient(hunting): authenticate will time out at 7608.436311s 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.215+0000 7f2fbd2a0640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f2fb80a98f0 0x7f2fb80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:44014/0 (socket says 172.21.15.42:44014) 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.215+0000 7f2fbd2a0640 1 -- 172.21.15.42:0/1562880348 learned_addr learned my addr 172.21.15.42:0/1562880348 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:53.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.239+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281137710 0 0) 0x7f2fb80b2a30 con 0x7f2fb80a98f0 2024-03-25T18:24:53.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.239+0000 7f2faffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.239+0000 7f2faffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.239+0000 7f2faffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.239+0000 7f2faffff640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:24:53.240 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.239+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa4003520 con 0x7f2fb80a98f0 2024-03-25T18:24:53.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.243+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041366159 0 0) 0x7f2fb80b28a0 con 0x7f2fb813f710 2024-03-25T18:24:53.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.243+0000 7f2faffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.243+0000 7f2faffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.243+0000 7f2faffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.243+0000 7f2faffff640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.243+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb80b2a30 con 0x7f2fb813f710 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.244+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760470383 0 0) 0x7f2fb8142520 con 0x7f2fb81430c0 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.244+0000 7f2faffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.244+0000 7f2faffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.244+0000 7f2faffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.245 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.244+0000 7f2faffff640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:24:53.246 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.244+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb80b28a0 con 0x7f2fb81430c0 2024-03-25T18:24:53.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 61933270 0 0) 0x7f2fa4003520 con 0x7f2fb80a98f0 2024-03-25T18:24:53.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 >> v1:172.21.15.120:6789/0 conn(0x7f2fb813f710 legacy=0x7f2fb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 >> v1:172.21.15.42:6789/0 conn(0x7f2fb81430c0 legacy=0x7f2fb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 1 monclient: found mon.noname-b 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb80b44c0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2fb4001690 con 0x7f2fb80a98f0 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2fbf504640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2fbf504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2faffff640 10 monclient: dump: 2024-03-25T18:24:53.248 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:53.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.246+0000 7f2fbf504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fb40017f0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2faffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2faffff640 1 -- 172.21.15.42:0/1562880348 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2fb4001c00 con 0x7f2fb80a98f0 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2fbf504640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2fbf504640 4 set_mon_vals no callback set 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2faffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2faffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.261+0000 7f2faffff640 10 monclient: dump: 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:53.263 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:53.264 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.262+0000 7f2fbf504640 10 monclient: shutdown 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.262+0000 7f2fbf504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.262+0000 7f2fbf504640 1 -- 172.21.15.42:0/1562880348 >> v1:172.21.15.42:6790/0 conn(0x7f2fb80a98f0 legacy=0x7f2fb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.262+0000 7f2fbf504640 1 -- 172.21.15.42:0/1562880348 shutdown_connections 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.262+0000 7f2fbf504640 1 -- 172.21.15.42:0/1562880348 >> 172.21.15.42:0/1562880348 conn(0x7f2fb80b7ea0 msgr2=0x7f2fb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.262+0000 7f2fbf504640 1 -- 172.21.15.42:0/1562880348 shutdown_connections 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.262+0000 7f2fbf504640 1 -- 172.21.15.42:0/1562880348 wait complete. 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.263+0000 7f2fbf504640 10 monclient: build_initial_monmap 2024-03-25T18:24:53.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.263+0000 7f2fbf504640 10 monclient: monmap: 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.263+0000 7f2fbf504640 1 librados: starting msgr at 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.263+0000 7f2fbf504640 1 librados: starting objecter 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 1 Processor -- start 2024-03-25T18:24:53.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 1 -- start start 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 1 librados: setting wanted keys 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 1 librados: calling monclient init 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient: init 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb80a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb81430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb81378c0 con 0x7f2fb813f710 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb8138460 con 0x7f2fb80a98f0 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb8138d30 con 0x7f2fb81430c0 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:53.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.264+0000 7f2fbf504640 10 monclient(hunting): authenticate will time out at 7608.491466s 2024-03-25T18:24:53.280 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.279+0000 7f2fbd2a0640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2fb80a98f0 0x7f2fb8137d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:48042/0 (socket says 172.21.15.42:48042) 2024-03-25T18:24:53.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.279+0000 7f2fbd2a0640 1 -- 172.21.15.42:0/2940321131 learned_addr learned my addr 172.21.15.42:0/2940321131 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:24:53.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.284+0000 7f2fadffb640 10 client.?.objecter ms_handle_connect 0x7f2fb80a98f0 2024-03-25T18:24:53.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.284+0000 7f2fadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.291 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.290+0000 7f2fadffb640 10 client.?.objecter ms_handle_connect 0x7f2fb81430c0 2024-03-25T18:24:53.291 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.290+0000 7f2fadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.297+0000 7f2fadffb640 10 client.?.objecter ms_handle_connect 0x7f2fb813f710 2024-03-25T18:24:53.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.297+0000 7f2fadffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.298+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720747970 0 0) 0x7f2fb8138460 con 0x7f2fb80a98f0 2024-03-25T18:24:53.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.298+0000 7f2fadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.298+0000 7f2fadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.298+0000 7f2fadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.298+0000 7f2fadffb640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:24:53.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.298+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c004eb0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.306+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116197196 0 0) 0x7f2fb8138d30 con 0x7f2fb81430c0 2024-03-25T18:24:53.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.306+0000 7f2fadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.306+0000 7f2fadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.306+0000 7f2fadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.306+0000 7f2fadffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:24:53.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.306+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb8138460 con 0x7f2fb81430c0 2024-03-25T18:24:53.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.313+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233616314 0 0) 0x7f2fb81378c0 con 0x7f2fb813f710 2024-03-25T18:24:53.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.313+0000 7f2fadffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.313+0000 7f2fadffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.313+0000 7f2fadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.313+0000 7f2fadffb640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:24:53.314 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.313+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb8138d30 con 0x7f2fb813f710 2024-03-25T18:24:53.322 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.321+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3735900661 0 0) 0x7f2f9c004eb0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.321+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb81378c0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.321+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2fb4001bc0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.321+0000 7f2fadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.321+0000 7f2fadffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.321+0000 7f2fadffb640 10 monclient(hunting): dump: 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.321+0000 7f2fadffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1981278131 0 0) 0x7f2fb81378c0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 >> v1:172.21.15.42:6790/0 conn(0x7f2fb81430c0 legacy=0x7f2fb81db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 >> v1:172.21.15.42:6789/0 conn(0x7f2fb813f710 legacy=0x7f2fb81db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 1 monclient: found mon.b 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb81dc070 con 0x7f2fb80a98f0 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 10 monclient: _renew_subs 2024-03-25T18:24:53.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb8138fa0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 10 monclient: _renew_subs 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb8139560 con 0x7f2fb80a98f0 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.342+0000 7f2fbf504640 1 librados: init done 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.343+0000 7f2fbf504640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:53.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.343+0000 7f2fbf504640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:53.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.343+0000 7f2fbf504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:53.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.343+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb80627b0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.618+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fb40011f0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.618+0000 7f2fadffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.618+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2fb4004cb0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.618+0000 7f2fadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.618+0000 7f2fadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.618+0000 7f2fadffb640 10 monclient: dump: 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:24:53.620 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.621 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:24:53.621 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:24:53.621 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:24:53.621 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:24:53.621 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:24:53.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.618+0000 7f2faf7fe640 4 set_mon_vals no callback set 2024-03-25T18:24:53.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.619+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f2fb401ad00 con 0x7f2fb80a98f0 2024-03-25T18:24:53.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.620+0000 7f2fbca9f640 10 monclient: get_auth_request con 0x7f2f9c06dd50 auth_method 0 2024-03-25T18:24:53.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.620+0000 7f2fadffb640 10 client.4465.objecter ms_handle_connect 0x7f2f9c06dd50 2024-03-25T18:24:53.626 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f2fb40549d0 con 0x7f2fb80a98f0 2024-03-25T18:24:53.626 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 10 client.4465.objecter ms_dispatch 0x7f2fb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:53.626 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:53.626 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:53.626 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T18:24:53.626 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fb4050310 con 0x7f2fb80a98f0 2024-03-25T18:24:53.626 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:53.627 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.625+0000 7f2fadffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:53.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.855+0000 7f2fbf504640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:53.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.856+0000 7f2fbf504640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:53.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.856+0000 7f2fbf504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:24:53.857 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.856+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fb813c790 con 0x7f2fb80a98f0 2024-03-25T18:24:53.894 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.892+0000 7f2fadffb640 1 -- 172.21.15.42:0/2940321131 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f2fb4052410 con 0x7f2fb80a98f0 2024-03-25T18:24:53.894 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.892+0000 7f2fadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:53.894 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.892+0000 7f2fadffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:53.894 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:24:53.894 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 10 librados: watch_flush enter 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 10 librados: watch_flush exit 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 10 client.4465.objecter successfully canceled tick 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 >> v1:172.21.15.42:6800/3081750888 conn(0x7f2f9c06dd50 legacy=0x7f2f9c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 10 monclient: shutdown 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 >> v1:172.21.15.120:6789/0 conn(0x7f2fb80a98f0 legacy=0x7f2fb8137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 shutdown_connections 2024-03-25T18:24:53.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 >> 172.21.15.42:0/2940321131 conn(0x7f2fb80b7ea0 msgr2=0x7f2fb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:53.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 shutdown_connections 2024-03-25T18:24:53.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.895+0000 7f2fbf504640 1 -- 172.21.15.42:0/2940321131 wait complete. 2024-03-25T18:24:53.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:24:53.896+0000 7f2fbf504640 1 librados: shutdown 2024-03-25T18:24:53.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:00.714 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:00.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.831+0000 7ff1889fe640 10 monclient: get_monmap_and_config 2024-03-25T18:25:00.832 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.831+0000 7ff1889fe640 10 monclient: build_initial_monmap 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.831+0000 7ff1889fe640 10 monclient: monmap: 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:00.832630+0000 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:00.832630+0000 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:00.833 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:00.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.832+0000 7ff1889fe640 1 Processor -- start 2024-03-25T18:25:00.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.832+0000 7ff1889fe640 1 -- start start 2024-03-25T18:25:00.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.832+0000 7ff1889fe640 10 monclient: init 2024-03-25T18:25:00.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:00.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:00.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff1840a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:00.834 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 10 monclient(hunting): picked mon.noname-b con 0x7ff18413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:00.835 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff1841430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:00.835 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff184142520 con 0x7ff1840a98f0 2024-03-25T18:25:00.835 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840b28a0 con 0x7ff1841430c0 2024-03-25T18:25:00.835 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840b2a30 con 0x7ff18413f710 2024-03-25T18:25:00.835 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:00.835 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.833+0000 7ff1889fe640 10 monclient(hunting): authenticate will time out at 7616.059883s 2024-03-25T18:25:00.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.843+0000 7ff18259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff1840a98f0 0x7ff1840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53574/0 (socket says 172.21.15.42:53574) 2024-03-25T18:25:00.845 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.843+0000 7ff18259c640 1 -- 172.21.15.42:0/1816243346 learned_addr learned my addr 172.21.15.42:0/1816243346 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.863+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630017942 0 0) 0x7ff1840b2a30 con 0x7ff18413f710 2024-03-25T18:25:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.863+0000 7ff181d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.863+0000 7ff181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.863+0000 7ff181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.863+0000 7ff181d9b640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:25:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.863+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff170003520 con 0x7ff18413f710 2024-03-25T18:25:00.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.867+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727738311 0 0) 0x7ff184142520 con 0x7ff1840a98f0 2024-03-25T18:25:00.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.867+0000 7ff181d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.867+0000 7ff181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.867+0000 7ff181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.869 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.867+0000 7ff181d9b640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:25:00.869 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.867+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1840b2a30 con 0x7ff1840a98f0 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689135955 0 0) 0x7ff1840b28a0 con 0x7ff1841430c0 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff184142520 con 0x7ff1841430c0 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993570696 0 0) 0x7ff170003520 con 0x7ff18413f710 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 >> v1:172.21.15.120:6789/0 conn(0x7ff1841430c0 legacy=0x7ff18414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 >> v1:172.21.15.42:6789/0 conn(0x7ff1840a98f0 legacy=0x7ff1840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 monclient: found mon.noname-b 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840b44c0 con 0x7ff18413f710 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient: _finish_auth 0 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff16c001690 con 0x7ff18413f710 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff181d9b640 10 monclient: dump: 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:00.871 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff1889fe640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:25:00.872 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.868+0000 7ff1889fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:00.878 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff16c001870 con 0x7ff18413f710 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff181d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff181d9b640 1 -- 172.21.15.42:0/1816243346 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff16c001c80 con 0x7ff18413f710 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff1889fe640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff1889fe640 4 set_mon_vals no callback set 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff181d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.877+0000 7ff181d9b640 10 monclient: dump: 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:00.879 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:00.880 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:00.880 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:00.880 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:00.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.878+0000 7ff1889fe640 10 monclient: shutdown 2024-03-25T18:25:00.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.878+0000 7ff1889fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:00.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.878+0000 7ff1889fe640 1 -- 172.21.15.42:0/1816243346 >> v1:172.21.15.42:6790/0 conn(0x7ff18413f710 legacy=0x7ff1841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.878+0000 7ff1889fe640 1 -- 172.21.15.42:0/1816243346 shutdown_connections 2024-03-25T18:25:00.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.878+0000 7ff1889fe640 1 -- 172.21.15.42:0/1816243346 >> 172.21.15.42:0/1816243346 conn(0x7ff1840b7ea0 msgr2=0x7ff1840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:00.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.878+0000 7ff1889fe640 1 -- 172.21.15.42:0/1816243346 shutdown_connections 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.878+0000 7ff1889fe640 1 -- 172.21.15.42:0/1816243346 wait complete. 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.879+0000 7ff1889fe640 10 monclient: build_initial_monmap 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.879+0000 7ff1889fe640 10 monclient: monmap: 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:00.882 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 1 librados: starting msgr at 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 1 librados: starting objecter 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 1 Processor -- start 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 1 -- start start 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 1 librados: setting wanted keys 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 1 librados: calling monclient init 2024-03-25T18:25:00.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 10 monclient: init 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 10 monclient(hunting): picked mon.noname-b con 0x7ff1840a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.880+0000 7ff1889fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff18413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.881+0000 7ff1889fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff1841430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.881+0000 7ff1889fe640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff184137df0 con 0x7ff1841430c0 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.881+0000 7ff1889fe640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1841337a0 con 0x7ff1840a98f0 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.881+0000 7ff1889fe640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff184133930 con 0x7ff18413f710 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.881+0000 7ff1889fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:00.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.881+0000 7ff1889fe640 10 monclient(hunting): authenticate will time out at 7616.107672s 2024-03-25T18:25:00.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.896+0000 7ff17bfff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ff18413f710 0x7ff1841376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:55992/0 (socket says 172.21.15.42:55992) 2024-03-25T18:25:00.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.896+0000 7ff17bfff640 1 -- 172.21.15.42:0/2277279082 learned_addr learned my addr 172.21.15.42:0/2277279082 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:00.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.897+0000 7ff17affd640 10 client.?.objecter ms_handle_connect 0x7ff18413f710 2024-03-25T18:25:00.898 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.897+0000 7ff17affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:00.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.902+0000 7ff17affd640 10 client.?.objecter ms_handle_connect 0x7ff1841430c0 2024-03-25T18:25:00.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.902+0000 7ff17affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:00.904 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.903+0000 7ff17affd640 10 client.?.objecter ms_handle_connect 0x7ff1840a98f0 2024-03-25T18:25:00.904 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.903+0000 7ff17affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:00.907 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.906+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032891631 0 0) 0x7ff184133930 con 0x7ff18413f710 2024-03-25T18:25:00.907 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.906+0000 7ff17affd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.907 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.906+0000 7ff17affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.907 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.906+0000 7ff17affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.908 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.906+0000 7ff17affd640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:25:00.908 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.906+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff164004eb0 con 0x7ff18413f710 2024-03-25T18:25:00.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.908+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945599066 0 0) 0x7ff184137df0 con 0x7ff1841430c0 2024-03-25T18:25:00.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.908+0000 7ff17affd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.908+0000 7ff17affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.908+0000 7ff17affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.909 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.908+0000 7ff17affd640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:25:00.910 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.908+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff184133930 con 0x7ff1841430c0 2024-03-25T18:25:00.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.914+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903000598 0 0) 0x7ff164004eb0 con 0x7ff18413f710 2024-03-25T18:25:00.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.914+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff184137df0 con 0x7ff18413f710 2024-03-25T18:25:00.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.914+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff16c001c40 con 0x7ff18413f710 2024-03-25T18:25:00.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.914+0000 7ff17affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.914+0000 7ff17affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.914+0000 7ff17affd640 10 monclient(hunting): dump: 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:00.916 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:00.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.914+0000 7ff17affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:00.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.916+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068560594 0 0) 0x7ff1841337a0 con 0x7ff1840a98f0 2024-03-25T18:25:00.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.916+0000 7ff17affd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.916+0000 7ff17affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.916+0000 7ff17affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.916+0000 7ff17affd640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:25:00.917 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.916+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff164004eb0 con 0x7ff1840a98f0 2024-03-25T18:25:00.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.923+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 43285221 0 0) 0x7ff184137df0 con 0x7ff18413f710 2024-03-25T18:25:00.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 >> v1:172.21.15.120:6789/0 conn(0x7ff1840a98f0 legacy=0x7ff1841392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 >> v1:172.21.15.42:6789/0 conn(0x7ff1841430c0 legacy=0x7ff184133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:00.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 1 monclient: found mon.c 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1841367d0 con 0x7ff18413f710 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 10 monclient: _finish_auth 0 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff17affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 5 monclient: authenticate success, global_id 4466 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 10 monclient: _renew_subs 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1841369e0 con 0x7ff18413f710 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 10 monclient: _renew_subs 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:00.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18413b640 con 0x7ff18413f710 2024-03-25T18:25:00.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.924+0000 7ff1889fe640 1 librados: init done 2024-03-25T18:25:00.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.925+0000 7ff1889fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:00.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.925+0000 7ff1889fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:00.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.925+0000 7ff1889fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:00.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:00.925+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1840627b0 con 0x7ff18413f710 2024-03-25T18:25:01.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.248+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff16c001210 con 0x7ff18413f710 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.248+0000 7ff17affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.248+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff16c003ba0 con 0x7ff18413f710 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.248+0000 7ff17affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.248+0000 7ff17affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.248+0000 7ff17affd640 10 monclient: dump: 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:01.250 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:01.251 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:01.251 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:01.251 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:01.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.248+0000 7ff18159a640 4 set_mon_vals no callback set 2024-03-25T18:25:01.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.249+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff16c003dc0 con 0x7ff18413f710 2024-03-25T18:25:01.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.250+0000 7ff18259c640 10 monclient: get_auth_request con 0x7ff16406de00 auth_method 0 2024-03-25T18:25:01.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.250+0000 7ff17affd640 10 client.4466.objecter ms_handle_connect 0x7ff16406de00 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.256+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7ff16c054960 con 0x7ff18413f710 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.256+0000 7ff17affd640 10 client.4466.objecter ms_dispatch 0x7ff184000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.256+0000 7ff17affd640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.256+0000 7ff17affd640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.256+0000 7ff17affd640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.257+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff16c050470 con 0x7ff18413f710 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.257+0000 7ff17affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:01.258 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.257+0000 7ff17affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:01.497 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.496+0000 7ff1889fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:01.497 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.496+0000 7ff1889fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:01.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.496+0000 7ff1889fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:01.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.496+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff184137300 con 0x7ff18413f710 2024-03-25T18:25:01.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.539+0000 7ff17affd640 1 -- 172.21.15.42:0/2277279082 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7ff16c0523a0 con 0x7ff18413f710 2024-03-25T18:25:01.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.539+0000 7ff17affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:01.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.539+0000 7ff17affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:01.540 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:01.541 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:01.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.541+0000 7ff1889fe640 10 librados: watch_flush enter 2024-03-25T18:25:01.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.541+0000 7ff1889fe640 10 librados: watch_flush exit 2024-03-25T18:25:01.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.541+0000 7ff1889fe640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T18:25:01.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.541+0000 7ff1889fe640 10 client.4466.objecter successfully canceled tick 2024-03-25T18:25:01.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff16406de00 legacy=0x7ff1640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 10 monclient: shutdown 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 >> v1:172.21.15.42:6790/0 conn(0x7ff18413f710 legacy=0x7ff1841376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 shutdown_connections 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 >> 172.21.15.42:0/2277279082 conn(0x7ff1840b7ea0 msgr2=0x7ff184074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 shutdown_connections 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 1 -- 172.21.15.42:0/2277279082 wait complete. 2024-03-25T18:25:01.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:01.542+0000 7ff1889fe640 1 librados: shutdown 2024-03-25T18:25:01.555 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:07.173 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:50.342 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.208 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:50.377 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.231 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:50.401 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.233 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:50.354 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.238 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:50.358 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.239 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:05:50.409 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.245 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:50.414 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.262 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:05:50.383 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:07.295 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:05:50.465 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:08.361 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.478+0000 7f6b604ff640 10 monclient: get_monmap_and_config 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.478+0000 7f6b604ff640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.478+0000 7f6b604ff640 10 monclient: monmap: 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:08.479331+0000 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:08.479331+0000 2024-03-25T18:25:08.479 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.479+0000 7f6b604ff640 1 Processor -- start 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.479+0000 7f6b604ff640 1 -- start start 2024-03-25T18:25:08.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.479+0000 7f6b604ff640 10 monclient: init 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 10 monclient(hunting): picked mon.noname-a con 0x7f6b5806adf0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 10 monclient(hunting): picked mon.noname-b con 0x7f6b580aee10 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 10 monclient(hunting): picked mon.noname-c con 0x7f6b58140860 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580af690 con 0x7f6b5806adf0 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580b0c70 con 0x7f6b58140860 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580b0f50 con 0x7f6b580aee10 2024-03-25T18:25:08.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.480+0000 7f6b604ff640 10 monclient(hunting): authenticate will time out at 7623.707015s 2024-03-25T18:25:08.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.489+0000 7f6b5e29b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6b5806adf0 0x7f6b5813d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50848/0 (socket says 172.21.15.42:50848) 2024-03-25T18:25:08.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.489+0000 7f6b5e29b640 1 -- 172.21.15.42:0/3443118895 learned_addr learned my addr 172.21.15.42:0/3443118895 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:08.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.506+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538683109 0 0) 0x7f6b580af690 con 0x7f6b5806adf0 2024-03-25T18:25:08.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.506+0000 7f6b5d299640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.506+0000 7f6b5d299640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.506+0000 7f6b5d299640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.506+0000 7f6b5d299640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:25:08.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.506+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b44003520 con 0x7f6b5806adf0 2024-03-25T18:25:08.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3423099417 0 0) 0x7f6b44003520 con 0x7f6b5806adf0 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 >> v1:172.21.15.42:6790/0 conn(0x7f6b580aee10 legacy=0x7f6b58140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 >> v1:172.21.15.120:6789/0 conn(0x7f6b58140860 legacy=0x7f6b5814b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 1 monclient: found mon.noname-a 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5813dc40 con 0x7f6b5806adf0 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b54001690 con 0x7f6b5806adf0 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:08.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b5d299640 10 monclient: dump: 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b604ff640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:25:08.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.515+0000 7f6b604ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b540017f0 con 0x7f6b5806adf0 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b5d299640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b5d299640 1 -- 172.21.15.42:0/3443118895 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b54001c00 con 0x7f6b5806adf0 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b604ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b604ff640 4 set_mon_vals no callback set 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b5d299640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b5d299640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.528+0000 7f6b5d299640 10 monclient: dump: 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.530 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.531 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.531 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:08.531 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:08.531 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:08.531 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:08.531 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.529+0000 7f6b604ff640 10 monclient: shutdown 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.529+0000 7f6b604ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.529+0000 7f6b604ff640 1 -- 172.21.15.42:0/3443118895 >> v1:172.21.15.42:6789/0 conn(0x7f6b5806adf0 legacy=0x7f6b5813d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.529+0000 7f6b604ff640 1 -- 172.21.15.42:0/3443118895 shutdown_connections 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.529+0000 7f6b604ff640 1 -- 172.21.15.42:0/3443118895 >> 172.21.15.42:0/3443118895 conn(0x7f6b580b4a80 msgr2=0x7f6b580b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.530+0000 7f6b604ff640 1 -- 172.21.15.42:0/3443118895 shutdown_connections 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.530+0000 7f6b604ff640 1 -- 172.21.15.42:0/3443118895 wait complete. 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.530+0000 7f6b604ff640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.530+0000 7f6b604ff640 10 monclient: monmap: 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.531+0000 7f6b604ff640 1 librados: starting msgr at 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.531+0000 7f6b604ff640 1 librados: starting objecter 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.531+0000 7f6b604ff640 1 Processor -- start 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.531+0000 7f6b604ff640 1 -- start start 2024-03-25T18:25:08.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.531+0000 7f6b604ff640 1 librados: setting wanted keys 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.531+0000 7f6b604ff640 1 librados: calling monclient init 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.531+0000 7f6b604ff640 10 monclient: init 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 10 monclient(hunting): picked mon.noname-a con 0x7f6b5806adf0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 10 monclient(hunting): picked mon.noname-b con 0x7f6b580aee10 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 10 monclient(hunting): picked mon.noname-c con 0x7f6b58131210 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b58070c50 con 0x7f6b5806adf0 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b58070970 con 0x7f6b580aee10 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b580aad00 con 0x7f6b58131210 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.532+0000 7f6b604ff640 10 monclient(hunting): authenticate will time out at 7623.758934s 2024-03-25T18:25:08.545 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.543+0000 7f6b5ea9c640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f6b58131210 0x7f6b581e2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:52468/0 (socket says 172.21.15.42:52468) 2024-03-25T18:25:08.545 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.543+0000 7f6b5ea9c640 1 -- 172.21.15.42:0/3603911249 learned_addr learned my addr 172.21.15.42:0/3603911249 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:08.546 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.545+0000 7f6b4effd640 10 client.?.objecter ms_handle_connect 0x7f6b58131210 2024-03-25T18:25:08.546 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.545+0000 7f6b4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.549 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.548+0000 7f6b4effd640 10 client.?.objecter ms_handle_connect 0x7f6b580aee10 2024-03-25T18:25:08.549 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.548+0000 7f6b4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.559+0000 7f6b4effd640 10 client.?.objecter ms_handle_connect 0x7f6b5806adf0 2024-03-25T18:25:08.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.559+0000 7f6b4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.561+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811808636 0 0) 0x7f6b580aad00 con 0x7f6b58131210 2024-03-25T18:25:08.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.561+0000 7f6b4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.561+0000 7f6b4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.561+0000 7f6b4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.561+0000 7f6b4effd640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:25:08.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.561+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c004eb0 con 0x7f6b58131210 2024-03-25T18:25:08.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.562+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656965609 0 0) 0x7f6b58070970 con 0x7f6b580aee10 2024-03-25T18:25:08.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.562+0000 7f6b4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.562+0000 7f6b4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.562+0000 7f6b4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.562+0000 7f6b4effd640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:25:08.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.562+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b580aad00 con 0x7f6b580aee10 2024-03-25T18:25:08.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.578+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4049639215 0 0) 0x7f6b580aad00 con 0x7f6b580aee10 2024-03-25T18:25:08.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.578+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b58070970 con 0x7f6b580aee10 2024-03-25T18:25:08.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.578+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b480018a0 con 0x7f6b580aee10 2024-03-25T18:25:08.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.578+0000 7f6b4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.578+0000 7f6b4effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:08.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.578+0000 7f6b4effd640 10 monclient(hunting): dump: 2024-03-25T18:25:08.579 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:08.580 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.578+0000 7f6b4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:08.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.584+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788055935 0 0) 0x7f6b58070c50 con 0x7f6b5806adf0 2024-03-25T18:25:08.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.584+0000 7f6b4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.584+0000 7f6b4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.584+0000 7f6b4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.584+0000 7f6b4effd640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:25:08.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.584+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b580aad00 con 0x7f6b5806adf0 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.591+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813998034 0 0) 0x7f6b3c004eb0 con 0x7f6b58131210 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.591+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b58070c50 con 0x7f6b58131210 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.591+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b50001c90 con 0x7f6b58131210 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.591+0000 7f6b4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.591+0000 7f6b4effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.591+0000 7f6b4effd640 10 monclient(hunting): dump: 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.592 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.593 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:08.593 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:08.593 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:08.593 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:08.593 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2926846357 0 0) 0x7f6b58070970 con 0x7f6b580aee10 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 >> v1:172.21.15.42:6790/0 conn(0x7f6b58131210 legacy=0x7f6b581e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 >> v1:172.21.15.42:6789/0 conn(0x7f6b5806adf0 legacy=0x7f6b5806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 1 monclient: found mon.b 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580aafe0 con 0x7f6b580aee10 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:25:08.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 10 monclient: _renew_subs 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b580ab2c0 con 0x7f6b580aee10 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 10 monclient: _renew_subs 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b580aaad0 con 0x7f6b580aee10 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.601+0000 7f6b604ff640 1 librados: init done 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.602+0000 7f6b604ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.602+0000 7f6b604ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.602+0000 7f6b604ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:08.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.602+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b580627b0 con 0x7f6b580aee10 2024-03-25T18:25:08.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.901+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b48001d50 con 0x7f6b580aee10 2024-03-25T18:25:08.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.901+0000 7f6b4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:08.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.901+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b480037d0 con 0x7f6b580aee10 2024-03-25T18:25:08.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.901+0000 7f6b4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.901+0000 7f6b4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:08.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.901+0000 7f6b4effd640 10 monclient: dump: 2024-03-25T18:25:08.903 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:08.904 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.901+0000 7f6b5ca98640 4 set_mon_vals no callback set 2024-03-25T18:25:08.905 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.902+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f6b480039f0 con 0x7f6b580aee10 2024-03-25T18:25:08.905 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.903+0000 7f6b5e29b640 10 monclient: get_auth_request con 0x7f6b3c06df70 auth_method 0 2024-03-25T18:25:08.905 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.903+0000 7f6b4effd640 10 client.4477.objecter ms_handle_connect 0x7f6b3c06df70 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.912+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f6b48053530 con 0x7f6b580aee10 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.912+0000 7f6b4effd640 10 client.4477.objecter ms_dispatch 0x7f6b58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.912+0000 7f6b4effd640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.912+0000 7f6b4effd640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.912+0000 7f6b4effd640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.913+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b4804ee70 con 0x7f6b580aee10 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.913+0000 7f6b4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:08.913+0000 7f6b4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:09.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.162+0000 7f6b604ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.162+0000 7f6b604ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.162+0000 7f6b604ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:09.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.162+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b5806b1d0 con 0x7f6b580aee10 2024-03-25T18:25:09.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.204+0000 7f6b4effd640 1 -- 172.21.15.42:0/3603911249 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f6b48050f70 con 0x7f6b580aee10 2024-03-25T18:25:09.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.204+0000 7f6b4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.204+0000 7f6b4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:09.206 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:09.206 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:09.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 10 librados: watch_flush enter 2024-03-25T18:25:09.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 10 librados: watch_flush exit 2024-03-25T18:25:09.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:25:09.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 >> v1:172.21.15.42:6800/3081750888 conn(0x7f6b3c06df70 legacy=0x7f6b3c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 10 monclient: shutdown 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 >> v1:172.21.15.120:6789/0 conn(0x7f6b580aee10 legacy=0x7f6b5806f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 shutdown_connections 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 >> 172.21.15.42:0/3603911249 conn(0x7f6b580b4a80 msgr2=0x7f6b5813d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 shutdown_connections 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 1 -- 172.21.15.42:0/3603911249 wait complete. 2024-03-25T18:25:09.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:09.207+0000 7f6b604ff640 1 librados: shutdown 2024-03-25T18:25:09.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:16.027 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:16.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.139+0000 7f6b9c09a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.139+0000 7f6b9c09a640 10 monclient: build_initial_monmap 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.139+0000 7f6b9c09a640 10 monclient: monmap: 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:16.140756+0000 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:16.140756+0000 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:16.141 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.140+0000 7f6b9c09a640 1 Processor -- start 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.140+0000 7f6b9c09a640 1 -- start start 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.140+0000 7f6b9c09a640 10 monclient: init 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 10 monclient(hunting): picked mon.noname-c con 0x7f6b940a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 10 monclient(hunting): picked mon.noname-b con 0x7f6b9413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 10 monclient(hunting): picked mon.noname-a con 0x7f6b941430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b94142520 con 0x7f6b941430c0 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b940b28a0 con 0x7f6b940a98f0 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b940b2a30 con 0x7f6b9413f710 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.141+0000 7f6b9c09a640 10 monclient(hunting): authenticate will time out at 7631.368480s 2024-03-25T18:25:16.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.153+0000 7f6b99e36640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f6b940a98f0 0x7f6b940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:54504/0 (socket says 172.21.15.42:54504) 2024-03-25T18:25:16.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.153+0000 7f6b99e36640 1 -- 172.21.15.42:0/1124267403 learned_addr learned my addr 172.21.15.42:0/1124267403 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:16.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.164+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060763119 0 0) 0x7f6b940b28a0 con 0x7f6b940a98f0 2024-03-25T18:25:16.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.164+0000 7f6b98e34640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.164+0000 7f6b98e34640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.164+0000 7f6b98e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.166 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.164+0000 7f6b98e34640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:25:16.166 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.164+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b80003520 con 0x7f6b940a98f0 2024-03-25T18:25:16.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.174+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256495192 0 0) 0x7f6b940b2a30 con 0x7f6b9413f710 2024-03-25T18:25:16.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.174+0000 7f6b98e34640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.174+0000 7f6b98e34640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.174+0000 7f6b98e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.174+0000 7f6b98e34640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:25:16.175 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.174+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b940b28a0 con 0x7f6b9413f710 2024-03-25T18:25:16.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3506926255 0 0) 0x7f6b80003520 con 0x7f6b940a98f0 2024-03-25T18:25:16.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 >> v1:172.21.15.42:6790/0 conn(0x7f6b9413f710 legacy=0x7f6b941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 >> v1:172.21.15.42:6789/0 conn(0x7f6b941430c0 legacy=0x7f6b9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:16.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 10 monclient: _finish_hunting 0 2024-03-25T18:25:16.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 1 monclient: found mon.noname-c 2024-03-25T18:25:16.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b940b44c0 con 0x7f6b940a98f0 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 10 monclient: _finish_auth 0 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 10 monclient: _check_auth_tickets 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b90001690 con 0x7f6b940a98f0 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b9c09a640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b9c09a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:16.179 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b98e34640 10 monclient: dump: 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:16.180 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.176+0000 7f6b9c09a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:16.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b90001870 con 0x7f6b940a98f0 2024-03-25T18:25:16.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b98e34640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:16.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b98e34640 1 -- 172.21.15.42:0/1124267403 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b90001c80 con 0x7f6b940a98f0 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b98e34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b98e34640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b98e34640 10 monclient: dump: 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:16.213 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:16.214 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:16.214 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b9c09a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:16.214 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.211+0000 7f6b9c09a640 4 set_mon_vals no callback set 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.212+0000 7f6b9c09a640 10 monclient: shutdown 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.212+0000 7f6b9c09a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.212+0000 7f6b9c09a640 1 -- 172.21.15.42:0/1124267403 >> v1:172.21.15.120:6789/0 conn(0x7f6b940a98f0 legacy=0x7f6b940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.212+0000 7f6b9c09a640 1 -- 172.21.15.42:0/1124267403 shutdown_connections 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.212+0000 7f6b9c09a640 1 -- 172.21.15.42:0/1124267403 >> 172.21.15.42:0/1124267403 conn(0x7f6b940b7ea0 msgr2=0x7f6b940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.212+0000 7f6b9c09a640 1 -- 172.21.15.42:0/1124267403 shutdown_connections 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.212+0000 7f6b9c09a640 1 -- 172.21.15.42:0/1124267403 wait complete. 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 10 monclient: build_initial_monmap 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 10 monclient: monmap: 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:16.216 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 1 librados: starting msgr at 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 1 librados: starting objecter 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 1 Processor -- start 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 1 -- start start 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 1 librados: setting wanted keys 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 1 librados: calling monclient init 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.213+0000 7f6b9c09a640 10 monclient: init 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:16.217 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 10 monclient(hunting): picked mon.noname-a con 0x7f6b940a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 10 monclient(hunting): picked mon.noname-b con 0x7f6b9413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 10 monclient(hunting): picked mon.noname-c con 0x7f6b941430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b941d7830 con 0x7f6b940a98f0 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b941d7500 con 0x7f6b9413f710 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b94138c70 con 0x7f6b941430c0 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.214+0000 7f6b9c09a640 10 monclient(hunting): authenticate will time out at 7631.440889s 2024-03-25T18:25:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.221+0000 7f6b9a637640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f6b941430c0 0x7f6b941e2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:57422/0 (socket says 172.21.15.42:57422) 2024-03-25T18:25:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.221+0000 7f6b9a637640 1 -- 172.21.15.42:0/3647960014 learned_addr learned my addr 172.21.15.42:0/3647960014 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:16.224 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.223+0000 7f6b8a7fc640 10 client.?.objecter ms_handle_connect 0x7f6b941430c0 2024-03-25T18:25:16.224 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.223+0000 7f6b8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.224 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.223+0000 7f6b8a7fc640 10 client.?.objecter ms_handle_connect 0x7f6b9413f710 2024-03-25T18:25:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.223+0000 7f6b8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.228+0000 7f6b8a7fc640 10 client.?.objecter ms_handle_connect 0x7f6b940a98f0 2024-03-25T18:25:16.230 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.228+0000 7f6b8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.230+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305335114 0 0) 0x7f6b94138c70 con 0x7f6b941430c0 2024-03-25T18:25:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.230+0000 7f6b8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.230+0000 7f6b8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.230+0000 7f6b8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.230+0000 7f6b8a7fc640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:25:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.230+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78004eb0 con 0x7f6b941430c0 2024-03-25T18:25:16.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.243+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299783310 0 0) 0x7f6b941d7500 con 0x7f6b9413f710 2024-03-25T18:25:16.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.243+0000 7f6b8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.243+0000 7f6b8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.243+0000 7f6b8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.243+0000 7f6b8a7fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:25:16.244 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.243+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b94138c70 con 0x7f6b9413f710 2024-03-25T18:25:16.246 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.245+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778380714 0 0) 0x7f6b941d7830 con 0x7f6b940a98f0 2024-03-25T18:25:16.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.245+0000 7f6b8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.245+0000 7f6b8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.245+0000 7f6b8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.245+0000 7f6b8a7fc640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:25:16.247 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.245+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b941d7500 con 0x7f6b940a98f0 2024-03-25T18:25:16.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.249+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1346607221 0 0) 0x7f6b78004eb0 con 0x7f6b941430c0 2024-03-25T18:25:16.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.249+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b941d7830 con 0x7f6b941430c0 2024-03-25T18:25:16.256 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.255+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1645870999 0 0) 0x7f6b94138c70 con 0x7f6b9413f710 2024-03-25T18:25:16.256 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.255+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b78004eb0 con 0x7f6b9413f710 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b8c001b90 con 0x7f6b941430c0 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.260 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2777215468 0 0) 0x7f6b941d7830 con 0x7f6b941430c0 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 >> v1:172.21.15.120:6789/0 conn(0x7f6b9413f710 legacy=0x7f6b94133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 >> v1:172.21.15.42:6789/0 conn(0x7f6b940a98f0 legacy=0x7f6b941d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 1 monclient: found mon.c 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94138e00 con 0x7f6b941430c0 2024-03-25T18:25:16.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b9c09a640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b9c09a640 10 monclient: _renew_subs 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b9c09a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.257+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b941346b0 con 0x7f6b941430c0 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.258+0000 7f6b9c09a640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.258+0000 7f6b9c09a640 10 monclient: _renew_subs 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.258+0000 7f6b9c09a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.258+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b94137fc0 con 0x7f6b941430c0 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.258+0000 7f6b9c09a640 1 librados: init done 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.258+0000 7f6b9c09a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.262 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.258+0000 7f6b9c09a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.259+0000 7f6b9c09a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:16.263 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.259+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b941d7690 con 0x7f6b941430c0 2024-03-25T18:25:16.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.475+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b8c002020 con 0x7f6b941430c0 2024-03-25T18:25:16.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.475+0000 7f6b8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:16.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.475+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6b8c004be0 con 0x7f6b941430c0 2024-03-25T18:25:16.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.475+0000 7f6b8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.475+0000 7f6b8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:16.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.475+0000 7f6b8a7fc640 10 monclient: dump: 2024-03-25T18:25:16.476 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:16.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.475+0000 7f6b8bfff640 4 set_mon_vals no callback set 2024-03-25T18:25:16.574 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.571+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f6b8c01abb0 con 0x7f6b941430c0 2024-03-25T18:25:16.574 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.571+0000 7f6b99e36640 10 monclient: get_auth_request con 0x7f6b7806dfe0 auth_method 0 2024-03-25T18:25:16.574 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.571+0000 7f6b8a7fc640 10 client.4478.objecter ms_handle_connect 0x7f6b7806dfe0 2024-03-25T18:25:16.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f6b8c054950 con 0x7f6b941430c0 2024-03-25T18:25:16.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 10 client.4478.objecter ms_dispatch 0x7f6b94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:16.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:16.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:16.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T18:25:16.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b8c050290 con 0x7f6b941430c0 2024-03-25T18:25:16.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.585+0000 7f6b8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:16.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.838+0000 7f6b9c09a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:16.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.838+0000 7f6b9c09a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:16.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.838+0000 7f6b9c09a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:25:16.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.838+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b941d79c0 con 0x7f6b941430c0 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.960+0000 7f6b8a7fc640 1 -- 172.21.15.42:0/3647960014 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f6b8c052390 con 0x7f6b941430c0 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.960+0000 7f6b8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.960+0000 7f6b8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 10 librados: watch_flush enter 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 10 librados: watch_flush exit 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 10 client.4478.objecter successfully canceled tick 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 >> v1:172.21.15.42:6800/3081750888 conn(0x7f6b7806dfe0 legacy=0x7f6b780704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 10 monclient: shutdown 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 >> v1:172.21.15.42:6790/0 conn(0x7f6b941430c0 legacy=0x7f6b941e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 shutdown_connections 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 >> 172.21.15.42:0/3647960014 conn(0x7f6b940b7ea0 msgr2=0x7f6b94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 shutdown_connections 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 1 -- 172.21.15.42:0/3647960014 wait complete. 2024-03-25T18:25:16.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:16.963+0000 7f6b9c09a640 1 librados: shutdown 2024-03-25T18:25:16.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:23.783 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:23.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.902+0000 7f700aeb3640 10 monclient: get_monmap_and_config 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.902+0000 7f700aeb3640 10 monclient: build_initial_monmap 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.902+0000 7f700aeb3640 10 monclient: monmap: 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:23.903862+0000 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:23.903862+0000 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:23.904 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:23.905 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.903+0000 7f700aeb3640 1 Processor -- start 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.903+0000 7f700aeb3640 1 -- start start 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.903+0000 7f700aeb3640 10 monclient: init 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 10 monclient(hunting): picked mon.noname-c con 0x7f70040a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 10 monclient(hunting): picked mon.noname-b con 0x7f700413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 10 monclient(hunting): picked mon.noname-a con 0x7f70041430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7004142520 con 0x7f70041430c0 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040b28a0 con 0x7f70040a98f0 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040b2a30 con 0x7f700413f710 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:23.906 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.904+0000 7f700aeb3640 10 monclient(hunting): authenticate will time out at 7639.131207s 2024-03-25T18:25:23.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.914+0000 7f7008c4f640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f70040a98f0 0x7f70040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:58888/0 (socket says 172.21.15.42:58888) 2024-03-25T18:25:23.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.914+0000 7f7008c4f640 1 -- 172.21.15.42:0/2861361947 learned_addr learned my addr 172.21.15.42:0/2861361947 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:23.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.931+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399178180 0 0) 0x7f70040b28a0 con 0x7f70040a98f0 2024-03-25T18:25:23.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.931+0000 7f70037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.931+0000 7f70037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.931+0000 7f70037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.931+0000 7f70037fe640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:25:23.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.931+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff4003520 con 0x7f70040a98f0 2024-03-25T18:25:23.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067828326 0 0) 0x7f70040b2a30 con 0x7f700413f710 2024-03-25T18:25:23.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:25:23.943 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70040b28a0 con 0x7f700413f710 2024-03-25T18:25:23.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716567556 0 0) 0x7f7004142520 con 0x7f70041430c0 2024-03-25T18:25:23.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:25:23.944 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.942+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70040b2a30 con 0x7f70041430c0 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944809654 0 0) 0x7f6ff4003520 con 0x7f70040a98f0 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 >> v1:172.21.15.42:6790/0 conn(0x7f700413f710 legacy=0x7f70041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 >> v1:172.21.15.42:6789/0 conn(0x7f70041430c0 legacy=0x7f700414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70040b44c0 con 0x7f70040a98f0 2024-03-25T18:25:23.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6ff8001690 con 0x7f70040a98f0 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.950+0000 7f70037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.951+0000 7f70037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.951+0000 7f70037fe640 10 monclient: dump: 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:23.953 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:23.954 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:23.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.951+0000 7f700aeb3640 5 monclient: authenticate success, global_id 4486 2024-03-25T18:25:23.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.951+0000 7f700aeb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ff80017f0 con 0x7f70040a98f0 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f70037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f70037fe640 1 -- 172.21.15.42:0/2861361947 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6ff8001c00 con 0x7f70040a98f0 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f700aeb3640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f700aeb3640 4 set_mon_vals no callback set 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f70037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f70037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:23.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f70037fe640 10 monclient: dump: 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:23.960 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:23.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f700aeb3640 10 monclient: shutdown 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f700aeb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.958+0000 7f700aeb3640 1 -- 172.21.15.42:0/2861361947 >> v1:172.21.15.120:6789/0 conn(0x7f70040a98f0 legacy=0x7f70040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.959+0000 7f700aeb3640 1 -- 172.21.15.42:0/2861361947 shutdown_connections 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.959+0000 7f700aeb3640 1 -- 172.21.15.42:0/2861361947 >> 172.21.15.42:0/2861361947 conn(0x7f70040b7ea0 msgr2=0x7f70040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.959+0000 7f700aeb3640 1 -- 172.21.15.42:0/2861361947 shutdown_connections 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.959+0000 7f700aeb3640 1 -- 172.21.15.42:0/2861361947 wait complete. 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 10 monclient: build_initial_monmap 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 10 monclient: monmap: 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:23.962 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 1 librados: starting msgr at 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 1 librados: starting objecter 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 1 Processor -- start 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 1 -- start start 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 1 librados: setting wanted keys 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 1 librados: calling monclient init 2024-03-25T18:25:23.963 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.960+0000 7f700aeb3640 10 monclient: init 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 10 monclient(hunting): picked mon.noname-b con 0x7f70040a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 10 monclient(hunting): picked mon.noname-a con 0x7f700413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 10 monclient(hunting): picked mon.noname-c con 0x7f70041430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7004139870 con 0x7f700413f710 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70041d51f0 con 0x7f70040a98f0 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7004138830 con 0x7f70041430c0 2024-03-25T18:25:23.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:23.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.961+0000 7f700aeb3640 10 monclient(hunting): authenticate will time out at 7639.188359s 2024-03-25T18:25:23.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.971+0000 7f7003fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f700413f710 0x7f70041d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56294/0 (socket says 172.21.15.42:56294) 2024-03-25T18:25:23.972 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.971+0000 7f7003fff640 1 -- 172.21.15.42:0/3843689573 learned_addr learned my addr 172.21.15.42:0/3843689573 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:23.973 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.972+0000 7f70017fa640 10 client.?.objecter ms_handle_connect 0x7f700413f710 2024-03-25T18:25:23.974 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.973+0000 7f70017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:23.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.984+0000 7f70017fa640 10 client.?.objecter ms_handle_connect 0x7f70040a98f0 2024-03-25T18:25:23.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.984+0000 7f70017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:23.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.986+0000 7f70017fa640 10 client.?.objecter ms_handle_connect 0x7f70041430c0 2024-03-25T18:25:23.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.986+0000 7f70017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:23.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.995+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456302146 0 0) 0x7f7004138830 con 0x7f70041430c0 2024-03-25T18:25:23.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.995+0000 7f70017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.995+0000 7f70017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.995+0000 7f70017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.995+0000 7f70017fa640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:25:23.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.995+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe8004eb0 con 0x7f70041430c0 2024-03-25T18:25:23.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.997+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979081741 0 0) 0x7f7004139870 con 0x7f700413f710 2024-03-25T18:25:23.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.997+0000 7f70017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.997+0000 7f70017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.997+0000 7f70017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.997+0000 7f70017fa640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:25:23.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:23.997+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004138830 con 0x7f700413f710 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.007+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4035941599 0 0) 0x7f7004138830 con 0x7f700413f710 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.007+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7004139870 con 0x7f700413f710 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.007+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6ff00019b0 con 0x7f700413f710 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.007+0000 7f70017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.007+0000 7f70017fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.007+0000 7f70017fa640 10 monclient(hunting): dump: 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:24.009 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.007+0000 7f70017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.009+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213081246 0 0) 0x7f70041d51f0 con 0x7f70040a98f0 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.009+0000 7f70017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.009+0000 7f70017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.009+0000 7f70017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.009+0000 7f70017fa640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:25:24.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.009+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004138830 con 0x7f70040a98f0 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.016+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3410751139 0 0) 0x7f6fe8004eb0 con 0x7f70041430c0 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.016+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe8005b00 con 0x7f70041430c0 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.016+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6ffc001bd0 con 0x7f70041430c0 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.016+0000 7f70017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.016+0000 7f70017fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.016+0000 7f70017fa640 10 monclient(hunting): dump: 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:24.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.021+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2742810875 0 0) 0x7f7004138830 con 0x7f70040a98f0 2024-03-25T18:25:24.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.021+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe8004eb0 con 0x7f70040a98f0 2024-03-25T18:25:24.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.021+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6ff8001b00 con 0x7f70040a98f0 2024-03-25T18:25:24.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.021+0000 7f70017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.021+0000 7f70017fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.021+0000 7f70017fa640 10 monclient(hunting): dump: 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:24.023 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:24.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1893490714 0 0) 0x7f7004139870 con 0x7f700413f710 2024-03-25T18:25:24.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 >> v1:172.21.15.42:6790/0 conn(0x7f70041430c0 legacy=0x7f70041e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 >> v1:172.21.15.120:6789/0 conn(0x7f70040a98f0 legacy=0x7f7004138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 1 monclient: found mon.a 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70041d77a0 con 0x7f700413f710 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f70017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 10 monclient: _renew_subs 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70041d83d0 con 0x7f700413f710 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:24.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 10 monclient: _renew_subs 2024-03-25T18:25:24.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:24.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7004138c80 con 0x7f700413f710 2024-03-25T18:25:24.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.022+0000 7f700aeb3640 1 librados: init done 2024-03-25T18:25:24.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.023+0000 7f700aeb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.023+0000 7f700aeb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.023+0000 7f700aeb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:24.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.023+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70041d79e0 con 0x7f700413f710 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.340+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ff0003610 con 0x7f700413f710 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.340+0000 7f70017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.340+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6ff0003840 con 0x7f700413f710 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.340+0000 7f70017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.340+0000 7f70017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.340+0000 7f70017fa640 10 monclient: dump: 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.342 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.340+0000 7f7002ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:24.343 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.341+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f6ff0003a60 con 0x7f700413f710 2024-03-25T18:25:24.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.342+0000 7f7008c4f640 10 monclient: get_auth_request con 0x7f6fe806e320 auth_method 0 2024-03-25T18:25:24.344 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.342+0000 7f70017fa640 10 client.4485.objecter ms_handle_connect 0x7f6fe806e320 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f6ff0054600 con 0x7f700413f710 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 10 client.4485.objecter ms_dispatch 0x7f7004000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ff0050110 con 0x7f700413f710 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.347+0000 7f70017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:24.578 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.577+0000 7f700aeb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.578 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.577+0000 7f700aeb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.578 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.577+0000 7f700aeb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:24.578 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.577+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7004074cc0 con 0x7f700413f710 2024-03-25T18:25:24.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.620+0000 7f70017fa640 1 -- 172.21.15.42:0/3843689573 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f6ff0052040 con 0x7f700413f710 2024-03-25T18:25:24.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.620+0000 7f70017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.620+0000 7f70017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:24.622 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:24.622 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:24.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.622+0000 7f700aeb3640 10 librados: watch_flush enter 2024-03-25T18:25:24.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.622+0000 7f700aeb3640 10 librados: watch_flush exit 2024-03-25T18:25:24.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 10 client.4485.objecter successfully canceled tick 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 >> v1:172.21.15.42:6800/3081750888 conn(0x7f6fe806e320 legacy=0x7f6fe80707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 10 monclient: shutdown 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 >> v1:172.21.15.42:6789/0 conn(0x7f700413f710 legacy=0x7f70041d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 shutdown_connections 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 >> 172.21.15.42:0/3843689573 conn(0x7f70040b7ea0 msgr2=0x7f7004074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 shutdown_connections 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 1 -- 172.21.15.42:0/3843689573 wait complete. 2024-03-25T18:25:24.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:24.623+0000 7f700aeb3640 1 librados: shutdown 2024-03-25T18:25:24.637 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:27.236 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:10.405 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.241 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:10.410 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.271 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:10.440 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.314 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:10.484 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.316 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:10.437 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.325 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:10.445 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.330 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:10.500 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.358 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:10.478 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:27.368 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:10.538 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:31.444 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:31.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.561+0000 7f184d855640 10 monclient: get_monmap_and_config 2024-03-25T18:25:31.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.561+0000 7f184d855640 10 monclient: build_initial_monmap 2024-03-25T18:25:31.561 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.561+0000 7f184d855640 10 monclient: monmap: 2024-03-25T18:25:31.561 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:31.561 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:31.561301+0000 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:31.561301+0000 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:31.562 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.561+0000 7f184d855640 1 Processor -- start 2024-03-25T18:25:31.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.561+0000 7f184d855640 1 -- start start 2024-03-25T18:25:31.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.561+0000 7f184d855640 10 monclient: init 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 10 monclient(hunting): picked mon.noname-a con 0x7f18480a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 10 monclient(hunting): picked mon.noname-c con 0x7f184813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 10 monclient(hunting): picked mon.noname-b con 0x7f18481430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1848142520 con 0x7f18480a98f0 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480b28a0 con 0x7f184813f710 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480b2a30 con 0x7f18481430c0 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:31.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.562+0000 7f184d855640 10 monclient(hunting): authenticate will time out at 7646.788332s 2024-03-25T18:25:31.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.570+0000 7f1846ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f18480a98f0 0x7f18480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43090/0 (socket says 172.21.15.42:43090) 2024-03-25T18:25:31.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.570+0000 7f1846ffd640 1 -- 172.21.15.42:0/1496210148 learned_addr learned my addr 172.21.15.42:0/1496210148 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:31.582 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.582+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551809196 0 0) 0x7f1848142520 con 0x7f18480a98f0 2024-03-25T18:25:31.582 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.582+0000 7f1845ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.582 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.582+0000 7f1845ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.582 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.582+0000 7f1845ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.582 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.582+0000 7f1845ffb640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:25:31.582 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.582+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1830003520 con 0x7f18480a98f0 2024-03-25T18:25:31.583 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.583+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137129554 0 0) 0x7f18480b28a0 con 0x7f184813f710 2024-03-25T18:25:31.583 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.583+0000 7f1845ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.583 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.583+0000 7f1845ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.583 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.583+0000 7f1845ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.583 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.583+0000 7f1845ffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:25:31.583 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.583+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848142520 con 0x7f184813f710 2024-03-25T18:25:31.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.585+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745939777 0 0) 0x7f18480b2a30 con 0x7f18481430c0 2024-03-25T18:25:31.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.585+0000 7f1845ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.585+0000 7f1845ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.585+0000 7f1845ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.585+0000 7f1845ffb640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:25:31.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.585+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18480b28a0 con 0x7f18481430c0 2024-03-25T18:25:31.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1558675094 0 0) 0x7f1830003520 con 0x7f18480a98f0 2024-03-25T18:25:31.589 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 >> v1:172.21.15.42:6790/0 conn(0x7f18481430c0 legacy=0x7f184814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 >> v1:172.21.15.120:6789/0 conn(0x7f184813f710 legacy=0x7f18481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 1 monclient: found mon.noname-a 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18480b44c0 con 0x7f18480a98f0 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:31.590 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f184d855640 5 monclient: authenticate success, global_id 4488 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f184d855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1838001650 con 0x7f18480a98f0 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f1845ffb640 10 monclient: dump: 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:31.591 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.592 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.589+0000 7f184d855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:31.598 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18380018a0 con 0x7f18480a98f0 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f1845ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f1845ffb640 1 -- 172.21.15.42:0/1496210148 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1838001c30 con 0x7f18480a98f0 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f1845ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f1845ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f1845ffb640 10 monclient: dump: 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:31.599 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:31.600 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f184d855640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:31.600 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.598+0000 7f184d855640 4 set_mon_vals no callback set 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.599+0000 7f184d855640 10 monclient: shutdown 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.599+0000 7f184d855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.599+0000 7f184d855640 1 -- 172.21.15.42:0/1496210148 >> v1:172.21.15.42:6789/0 conn(0x7f18480a98f0 legacy=0x7f18480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.599+0000 7f184d855640 1 -- 172.21.15.42:0/1496210148 shutdown_connections 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.599+0000 7f184d855640 1 -- 172.21.15.42:0/1496210148 >> 172.21.15.42:0/1496210148 conn(0x7f18480b7ea0 msgr2=0x7f18480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.599+0000 7f184d855640 1 -- 172.21.15.42:0/1496210148 shutdown_connections 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.599+0000 7f184d855640 1 -- 172.21.15.42:0/1496210148 wait complete. 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.600+0000 7f184d855640 10 monclient: build_initial_monmap 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.600+0000 7f184d855640 10 monclient: monmap: 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:31.602 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.600+0000 7f184d855640 1 librados: starting msgr at 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.600+0000 7f184d855640 1 librados: starting objecter 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.600+0000 7f184d855640 1 Processor -- start 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 1 -- start start 2024-03-25T18:25:31.603 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 1 librados: setting wanted keys 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 1 librados: calling monclient init 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient: init 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient(hunting): picked mon.noname-b con 0x7f18480a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient(hunting): picked mon.noname-c con 0x7f184813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient(hunting): picked mon.noname-a con 0x7f18481430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18481d7810 con 0x7f18481430c0 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1848137c70 con 0x7f18480a98f0 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18481d8440 con 0x7f184813f710 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:31.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.601+0000 7f184d855640 10 monclient(hunting): authenticate will time out at 7646.827441s 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.610+0000 7f1846ffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f18480a98f0 0x7f18481d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:58558/0 (socket says 172.21.15.42:58558) 2024-03-25T18:25:31.611 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.610+0000 7f1846ffd640 1 -- 172.21.15.42:0/3091963512 learned_addr learned my addr 172.21.15.42:0/3091963512 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:31.611 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.611+0000 7f182ffff640 10 client.?.objecter ms_handle_connect 0x7f18480a98f0 2024-03-25T18:25:31.611 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.611+0000 7f182ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.621+0000 7f182ffff640 10 client.?.objecter ms_handle_connect 0x7f18481430c0 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.621+0000 7f182ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.622+0000 7f182ffff640 10 client.?.objecter ms_handle_connect 0x7f184813f710 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.622+0000 7f182ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.629+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195071376 0 0) 0x7f1848137c70 con 0x7f18480a98f0 2024-03-25T18:25:31.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.629+0000 7f182ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.629+0000 7f182ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.629+0000 7f182ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.629+0000 7f182ffff640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:25:31.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.629+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834004eb0 con 0x7f18480a98f0 2024-03-25T18:25:31.635 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.635+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203982171 0 0) 0x7f18481d7810 con 0x7f18481430c0 2024-03-25T18:25:31.635 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.635+0000 7f182ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.635 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.635+0000 7f182ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.635 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.635+0000 7f182ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.635 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.635+0000 7f182ffff640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:25:31.635 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.635+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848137c70 con 0x7f18481430c0 2024-03-25T18:25:31.639 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.639+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372526017 0 0) 0x7f18481d8440 con 0x7f184813f710 2024-03-25T18:25:31.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.639+0000 7f182ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.639+0000 7f182ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.639+0000 7f182ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.639+0000 7f182ffff640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:25:31.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.639+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18481d7810 con 0x7f184813f710 2024-03-25T18:25:31.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.650+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3407881351 0 0) 0x7f1834004eb0 con 0x7f18480a98f0 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.650+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18481d8440 con 0x7f18480a98f0 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.650+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1838001bb0 con 0x7f18480a98f0 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.650+0000 7f182ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.650+0000 7f182ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.650+0000 7f182ffff640 10 monclient(hunting): dump: 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.651 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:31.652 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:31.652 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:31.652 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:31.652 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.650+0000 7f182ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.660+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230869040 0 0) 0x7f1848137c70 con 0x7f18481430c0 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.660+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1834004eb0 con 0x7f18481430c0 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.660+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1840001b50 con 0x7f18481430c0 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.660+0000 7f182ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.660+0000 7f182ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.660+0000 7f182ffff640 10 monclient(hunting): dump: 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:31.660 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:31.661 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.661+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4176090701 0 0) 0x7f18481d7810 con 0x7f184813f710 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.661+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1848137c70 con 0x7f184813f710 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.661+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f183c0018b0 con 0x7f184813f710 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.661+0000 7f182ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.661+0000 7f182ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.661+0000 7f182ffff640 10 monclient(hunting): dump: 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:31.662 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:31.663 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:31.663 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:31.663 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 530570124 0 0) 0x7f18481d8440 con 0x7f18480a98f0 2024-03-25T18:25:31.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 >> v1:172.21.15.42:6790/0 conn(0x7f184813f710 legacy=0x7f1848138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 >> v1:172.21.15.42:6789/0 conn(0x7f18481430c0 legacy=0x7f18481d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:31.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 1 monclient: found mon.b 2024-03-25T18:25:31.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:31.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18481d85d0 con 0x7f18480a98f0 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f182ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 10 monclient: _renew_subs 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18481d41e0 con 0x7f18480a98f0 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 10 monclient: _renew_subs 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1848139040 con 0x7f18480a98f0 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.662+0000 7f184d855640 1 librados: init done 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.663+0000 7f184d855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.663+0000 7f184d855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.663+0000 7f184d855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:31.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.663+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18480ac0d0 con 0x7f18480a98f0 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.680+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1838004bf0 con 0x7f18480a98f0 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.680+0000 7f182ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.680+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1838004f00 con 0x7f18480a98f0 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.680+0000 7f182ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.680+0000 7f182ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.680+0000 7f182ffff640 10 monclient: dump: 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:31.680 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:31.681 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.680+0000 7f18457fa640 4 set_mon_vals no callback set 2024-03-25T18:25:31.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.681+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f183801aed0 con 0x7f18480a98f0 2024-03-25T18:25:31.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.681+0000 7f18467fc640 10 monclient: get_auth_request con 0x7f183406e060 auth_method 0 2024-03-25T18:25:31.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.682+0000 7f182ffff640 10 client.4495.objecter ms_handle_connect 0x7f183406e060 2024-03-25T18:25:31.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.683+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f18380549e0 con 0x7f18480a98f0 2024-03-25T18:25:31.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.683+0000 7f182ffff640 10 client.4495.objecter ms_dispatch 0x7f1848000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:31.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.683+0000 7f182ffff640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:31.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.683+0000 7f182ffff640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:31.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.683+0000 7f182ffff640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T18:25:31.947 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.947+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1838050320 con 0x7f18480a98f0 2024-03-25T18:25:31.947 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.947+0000 7f182ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.947 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:31.947+0000 7f182ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:32.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.172+0000 7f184d855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.172+0000 7f184d855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.172+0000 7f184d855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:25:32.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.172+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1848136b50 con 0x7f18480a98f0 2024-03-25T18:25:32.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.197+0000 7f182ffff640 1 -- 172.21.15.42:0/3091963512 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f1848136b50 con 0x7f18480a98f0 2024-03-25T18:25:32.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.197+0000 7f182ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.197+0000 7f182ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:32.198 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:32.199 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 10 librados: watch_flush enter 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 10 librados: watch_flush exit 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 10 client.4495.objecter successfully canceled tick 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 >> v1:172.21.15.42:6800/3081750888 conn(0x7f183406e060 legacy=0x7f1834070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 10 monclient: shutdown 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 >> v1:172.21.15.120:6789/0 conn(0x7f18480a98f0 legacy=0x7f18481d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 shutdown_connections 2024-03-25T18:25:32.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 >> 172.21.15.42:0/3091963512 conn(0x7f18480b7ea0 msgr2=0x7f1848074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:32.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.199+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 shutdown_connections 2024-03-25T18:25:32.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.200+0000 7f184d855640 1 -- 172.21.15.42:0/3091963512 wait complete. 2024-03-25T18:25:32.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:32.200+0000 7f184d855640 1 librados: shutdown 2024-03-25T18:25:32.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:39.020 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:39.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.134+0000 7f287e913640 10 monclient: get_monmap_and_config 2024-03-25T18:25:39.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.134+0000 7f287e913640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.134+0000 7f287e913640 10 monclient: monmap: 2024-03-25T18:25:39.135 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:39.135 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:39.135237+0000 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:39.135237+0000 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:39.136 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:39.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.135+0000 7f287e913640 1 Processor -- start 2024-03-25T18:25:39.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.135+0000 7f287e913640 1 -- start start 2024-03-25T18:25:39.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.135+0000 7f287e913640 10 monclient: init 2024-03-25T18:25:39.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 10 monclient(hunting): picked mon.noname-a con 0x7f28780a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 10 monclient(hunting): picked mon.noname-b con 0x7f287813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 10 monclient(hunting): picked mon.noname-c con 0x7f28781430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2878142520 con 0x7f28780a98f0 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28780b28a0 con 0x7f28781430c0 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28780b2a30 con 0x7f287813f710 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.136+0000 7f287e913640 10 monclient(hunting): authenticate will time out at 7654.362863s 2024-03-25T18:25:39.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.148+0000 7f28777fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f287813f710 0x7f28781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:36024/0 (socket says 172.21.15.42:36024) 2024-03-25T18:25:39.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.148+0000 7f28777fe640 1 -- 172.21.15.42:0/1773382844 learned_addr learned my addr 172.21.15.42:0/1773382844 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:39.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.163+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744038996 0 0) 0x7f28780b2a30 con 0x7f287813f710 2024-03-25T18:25:39.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.163+0000 7f2876ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.163+0000 7f2876ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.163+0000 7f2876ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.163+0000 7f2876ffd640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:25:39.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.163+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2860003520 con 0x7f287813f710 2024-03-25T18:25:39.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.166+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793936743 0 0) 0x7f28780b28a0 con 0x7f28781430c0 2024-03-25T18:25:39.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.166+0000 7f2876ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.166+0000 7f2876ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.166+0000 7f2876ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.166+0000 7f2876ffd640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:25:39.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.166+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28780b2a30 con 0x7f28781430c0 2024-03-25T18:25:39.180 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2230437954 0 0) 0x7f28780b2a30 con 0x7f28781430c0 2024-03-25T18:25:39.180 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 >> v1:172.21.15.42:6790/0 conn(0x7f287813f710 legacy=0x7f28781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 >> v1:172.21.15.42:6789/0 conn(0x7f28780a98f0 legacy=0x7f28780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 1 monclient: found mon.noname-c 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780b44c0 con 0x7f28781430c0 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f28700019f0 con 0x7f28781430c0 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f2876ffd640 10 monclient: dump: 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:39.181 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f287e913640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:25:39.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.179+0000 7f287e913640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.183+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2870001b50 con 0x7f28781430c0 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.183+0000 7f2876ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.183+0000 7f2876ffd640 1 -- 172.21.15.42:0/1773382844 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f2870001fa0 con 0x7f28781430c0 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f2876ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f2876ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f2876ffd640 10 monclient: dump: 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.185 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.186 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:39.186 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:39.186 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:39.186 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:39.186 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:39.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f287e913640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:39.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f287e913640 4 set_mon_vals no callback set 2024-03-25T18:25:39.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f287e913640 10 monclient: shutdown 2024-03-25T18:25:39.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f287e913640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:39.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f287e913640 1 -- 172.21.15.42:0/1773382844 >> v1:172.21.15.120:6789/0 conn(0x7f28781430c0 legacy=0x7f287814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f287e913640 1 -- 172.21.15.42:0/1773382844 shutdown_connections 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.184+0000 7f287e913640 1 -- 172.21.15.42:0/1773382844 >> 172.21.15.42:0/1773382844 conn(0x7f28780b7ea0 msgr2=0x7f28780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.185+0000 7f287e913640 1 -- 172.21.15.42:0/1773382844 shutdown_connections 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.185+0000 7f287e913640 1 -- 172.21.15.42:0/1773382844 wait complete. 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.185+0000 7f287e913640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.185+0000 7f287e913640 10 monclient: monmap: 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:39.188 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.186+0000 7f287e913640 1 librados: starting msgr at 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.186+0000 7f287e913640 1 librados: starting objecter 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.186+0000 7f287e913640 1 Processor -- start 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.186+0000 7f287e913640 1 -- start start 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.186+0000 7f287e913640 1 librados: setting wanted keys 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.186+0000 7f287e913640 1 librados: calling monclient init 2024-03-25T18:25:39.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.186+0000 7f287e913640 10 monclient: init 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 10 monclient(hunting): picked mon.noname-c con 0x7f28780a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 10 monclient(hunting): picked mon.noname-a con 0x7f287813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 10 monclient(hunting): picked mon.noname-b con 0x7f28781430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2878137e20 con 0x7f287813f710 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28780b1720 con 0x7f28781430c0 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2878138710 con 0x7f28780a98f0 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.187+0000 7f287e913640 10 monclient(hunting): authenticate will time out at 7654.413928s 2024-03-25T18:25:39.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.192+0000 7f2877fff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f28780a98f0 0x7f2878137710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:36072/0 (socket says 172.21.15.42:36072) 2024-03-25T18:25:39.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.192+0000 7f2877fff640 1 -- 172.21.15.42:0/299034135 learned_addr learned my addr 172.21.15.42:0/299034135 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:39.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.195+0000 7f2874ff9640 10 client.?.objecter ms_handle_connect 0x7f28780a98f0 2024-03-25T18:25:39.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.195+0000 7f2874ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.210+0000 7f2874ff9640 10 client.?.objecter ms_handle_connect 0x7f28781430c0 2024-03-25T18:25:39.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.210+0000 7f2874ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.211+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998312167 0 0) 0x7f2878138710 con 0x7f28780a98f0 2024-03-25T18:25:39.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.212+0000 7f2874ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.212+0000 7f2874ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.212+0000 7f2874ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.212+0000 7f2874ff9640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:25:39.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.212+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2864004eb0 con 0x7f28780a98f0 2024-03-25T18:25:39.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.217+0000 7f2874ff9640 10 client.?.objecter ms_handle_connect 0x7f287813f710 2024-03-25T18:25:39.218 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.217+0000 7f2874ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.231+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816783565 0 0) 0x7f2878137e20 con 0x7f287813f710 2024-03-25T18:25:39.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.231+0000 7f2874ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.231+0000 7f2874ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.231+0000 7f2874ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.231+0000 7f2874ff9640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:25:39.232 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.231+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2878138710 con 0x7f287813f710 2024-03-25T18:25:39.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.296+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3381583990 0 0) 0x7f2878138710 con 0x7f287813f710 2024-03-25T18:25:39.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.296+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2878137e20 con 0x7f287813f710 2024-03-25T18:25:39.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.296+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f286c001950 con 0x7f287813f710 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.296+0000 7f2874ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.296+0000 7f2874ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.296+0000 7f2874ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.299 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.300 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:39.300 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:39.300 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:39.300 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:39.300 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:39.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.297+0000 7f2874ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:39.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.307+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770385006 0 0) 0x7f28780b1720 con 0x7f28781430c0 2024-03-25T18:25:39.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.307+0000 7f2874ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.307+0000 7f2874ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.307+0000 7f2874ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.307+0000 7f2874ff9640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:25:39.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.307+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2878138710 con 0x7f28781430c0 2024-03-25T18:25:39.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4246236847 0 0) 0x7f2878137e20 con 0x7f287813f710 2024-03-25T18:25:39.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 >> v1:172.21.15.42:6790/0 conn(0x7f28780a98f0 legacy=0x7f2878137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 >> v1:172.21.15.120:6789/0 conn(0x7f28781430c0 legacy=0x7f28781daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 1 monclient: found mon.a 2024-03-25T18:25:39.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:39.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2878138040 con 0x7f287813f710 2024-03-25T18:25:39.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f2874ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 5 monclient: authenticate success, global_id 4497 2024-03-25T18:25:39.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 10 monclient: _renew_subs 2024-03-25T18:25:39.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:39.320 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28781382f0 con 0x7f287813f710 2024-03-25T18:25:39.320 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:39.320 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 10 monclient: _renew_subs 2024-03-25T18:25:39.320 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:39.320 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28781e2fb0 con 0x7f287813f710 2024-03-25T18:25:39.320 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.315+0000 7f287e913640 1 librados: init done 2024-03-25T18:25:39.320 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.316+0000 7f287e913640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.316+0000 7f287e913640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.316+0000 7f287e913640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:39.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.316+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28780b3ef0 con 0x7f287813f710 2024-03-25T18:25:39.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f286c001d60 con 0x7f287813f710 2024-03-25T18:25:39.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f2874ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:39.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f286c004980 con 0x7f287813f710 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f2874ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f2874ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f2874ff9640 10 monclient: dump: 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:39.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f28767fc640 4 set_mon_vals no callback set 2024-03-25T18:25:39.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.728+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f286c004ba0 con 0x7f287813f710 2024-03-25T18:25:39.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.729+0000 7f2877fff640 10 monclient: get_auth_request con 0x7f286406dc40 auth_method 0 2024-03-25T18:25:39.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.729+0000 7f2874ff9640 10 client.4497.objecter ms_handle_connect 0x7f286406dc40 2024-03-25T18:25:39.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f286c0546b0 con 0x7f287813f710 2024-03-25T18:25:39.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 10 client.4497.objecter ms_dispatch 0x7f2878000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:39.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:39.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:39.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T18:25:39.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f286c0501c0 con 0x7f287813f710 2024-03-25T18:25:39.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.735+0000 7f2874ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:39.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.997+0000 7f287e913640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:39.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.997+0000 7f287e913640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:39.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.997+0000 7f287e913640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:39.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:39.998+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28780627b0 con 0x7f287813f710 2024-03-25T18:25:40.036 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.035+0000 7f2874ff9640 1 -- 172.21.15.42:0/299034135 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f286c0520f0 con 0x7f287813f710 2024-03-25T18:25:40.036 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.035+0000 7f2874ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:40.036 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.035+0000 7f2874ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:40.036 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:40.036 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:40.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 10 librados: watch_flush enter 2024-03-25T18:25:40.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 10 librados: watch_flush exit 2024-03-25T18:25:40.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T18:25:40.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 10 client.4497.objecter successfully canceled tick 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 >> v1:172.21.15.42:6800/3081750888 conn(0x7f286406dc40 legacy=0x7f2864070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 10 monclient: shutdown 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 >> v1:172.21.15.42:6789/0 conn(0x7f287813f710 legacy=0x7f28781da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 shutdown_connections 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.037+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 >> 172.21.15.42:0/299034135 conn(0x7f28780b7ea0 msgr2=0x7f2878074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.038+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 shutdown_connections 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.038+0000 7f287e913640 1 -- 172.21.15.42:0/299034135 wait complete. 2024-03-25T18:25:40.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:40.038+0000 7f287e913640 1 librados: shutdown 2024-03-25T18:25:40.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:46.856 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:46.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.977+0000 7f75748ba640 10 monclient: get_monmap_and_config 2024-03-25T18:25:46.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.977+0000 7f75748ba640 10 monclient: build_initial_monmap 2024-03-25T18:25:46.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.977+0000 7f75748ba640 10 monclient: monmap: 2024-03-25T18:25:46.978 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:46.978 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:46.978 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:46.978244+0000 2024-03-25T18:25:46.978 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:46.978244+0000 2024-03-25T18:25:46.979 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:46.979 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:46.979 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:46.979 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:46.979 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:46.979 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.977+0000 7f75748ba640 1 Processor -- start 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.977+0000 7f75748ba640 1 -- start start 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.977+0000 7f75748ba640 10 monclient: init 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 10 monclient(hunting): picked mon.noname-b con 0x7f7570061830 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 10 monclient(hunting): picked mon.noname-c con 0x7f757013c8c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 10 monclient(hunting): picked mon.noname-a con 0x7f7570072aa0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7570071d40 con 0x7f7570072aa0 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7570072ef0 con 0x7f757013c8c0 2024-03-25T18:25:46.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7570071f70 con 0x7f7570061830 2024-03-25T18:25:46.981 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:46.981 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.978+0000 7f75748ba640 10 monclient(hunting): authenticate will time out at 7662.205432s 2024-03-25T18:25:46.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.986+0000 7f756e59c640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f7570061830 0x7f757013c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:40772/0 (socket says 172.21.15.42:40772) 2024-03-25T18:25:46.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.986+0000 7f756e59c640 1 -- 172.21.15.42:0/1100526778 learned_addr learned my addr 172.21.15.42:0/1100526778 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:46.999 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.998+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978497801 0 0) 0x7f7570071f70 con 0x7f7570061830 2024-03-25T18:25:47.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.998+0000 7f756d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.998+0000 7f756d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.998+0000 7f756d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.998+0000 7f756d59a640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:25:47.000 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:46.998+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7558003520 con 0x7f7570061830 2024-03-25T18:25:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.011+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733081253 0 0) 0x7f7570072ef0 con 0x7f757013c8c0 2024-03-25T18:25:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.011+0000 7f756d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.011+0000 7f756d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.011+0000 7f756d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.011+0000 7f756d59a640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:25:47.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.011+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7570071f70 con 0x7f757013c8c0 2024-03-25T18:25:47.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.014+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385838779 0 0) 0x7f7570071d40 con 0x7f7570072aa0 2024-03-25T18:25:47.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.014+0000 7f756d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.014+0000 7f756d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.014+0000 7f756d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.014+0000 7f756d59a640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:25:47.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.014+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7570072ef0 con 0x7f7570072aa0 2024-03-25T18:25:47.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556232623 0 0) 0x7f7558003520 con 0x7f7570061830 2024-03-25T18:25:47.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 >> v1:172.21.15.120:6789/0 conn(0x7f757013c8c0 legacy=0x7f7570072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 >> v1:172.21.15.42:6789/0 conn(0x7f7570072aa0 legacy=0x7f757014a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 1 monclient: found mon.noname-b 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757006fbb0 con 0x7f7570061830 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 10 monclient: _finish_auth 0 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7560001690 con 0x7f7570061830 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f75748ba640 5 monclient: authenticate success, global_id 4499 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f75748ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:47.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f756d59a640 10 monclient: dump: 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:47.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.021+0000 7f75748ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75600017f0 con 0x7f7570061830 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f756d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f756d59a640 1 -- 172.21.15.42:0/1100526778 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7560001c00 con 0x7f7570061830 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f75748ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f75748ba640 4 set_mon_vals no callback set 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f756d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f756d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f756d59a640 10 monclient: dump: 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.037 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.038 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.038 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:47.038 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:47.038 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:47.038 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:47.038 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:47.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f75748ba640 10 monclient: shutdown 2024-03-25T18:25:47.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f75748ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:47.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.036+0000 7f75748ba640 1 -- 172.21.15.42:0/1100526778 >> v1:172.21.15.42:6790/0 conn(0x7f7570061830 legacy=0x7f757013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.037+0000 7f75748ba640 1 -- 172.21.15.42:0/1100526778 shutdown_connections 2024-03-25T18:25:47.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.037+0000 7f75748ba640 1 -- 172.21.15.42:0/1100526778 >> 172.21.15.42:0/1100526778 conn(0x7f75700b0ef0 msgr2=0x7f75700b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:47.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.037+0000 7f75748ba640 1 -- 172.21.15.42:0/1100526778 shutdown_connections 2024-03-25T18:25:47.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.037+0000 7f75748ba640 1 -- 172.21.15.42:0/1100526778 wait complete. 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.038+0000 7f75748ba640 10 monclient: build_initial_monmap 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.038+0000 7f75748ba640 10 monclient: monmap: 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:47.040 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.038+0000 7f75748ba640 1 librados: starting msgr at 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.038+0000 7f75748ba640 1 librados: starting objecter 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.038+0000 7f75748ba640 1 Processor -- start 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 1 -- start start 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 1 librados: setting wanted keys 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 1 librados: calling monclient init 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient: init 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient(hunting): picked mon.noname-a con 0x7f7570061830 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient(hunting): picked mon.noname-c con 0x7f7570072aa0 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:47.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient(hunting): picked mon.noname-b con 0x7f757013c8c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:47.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75700b09f0 con 0x7f7570061830 2024-03-25T18:25:47.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75701c6c50 con 0x7f757013c8c0 2024-03-25T18:25:47.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75700b05c0 con 0x7f7570072aa0 2024-03-25T18:25:47.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:47.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.039+0000 7f75748ba640 10 monclient(hunting): authenticate will time out at 7662.266407s 2024-03-25T18:25:47.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.045+0000 7f756e59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7570061830 0x7f75701c73c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59626/0 (socket says 172.21.15.42:59626) 2024-03-25T18:25:47.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.045+0000 7f756e59c640 1 -- 172.21.15.42:0/4025466498 learned_addr learned my addr 172.21.15.42:0/4025466498 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:47.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.046+0000 7f755effd640 10 client.?.objecter ms_handle_connect 0x7f7570061830 2024-03-25T18:25:47.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.046+0000 7f755effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.056+0000 7f755effd640 10 client.?.objecter ms_handle_connect 0x7f7570072aa0 2024-03-25T18:25:47.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.056+0000 7f755effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.058+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629982660 0 0) 0x7f75700b09f0 con 0x7f7570061830 2024-03-25T18:25:47.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.058+0000 7f755effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.058+0000 7f755effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.058+0000 7f755effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.058+0000 7f755effd640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:25:47.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.058+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550004eb0 con 0x7f7570061830 2024-03-25T18:25:47.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.062+0000 7f755effd640 10 client.?.objecter ms_handle_connect 0x7f757013c8c0 2024-03-25T18:25:47.064 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.062+0000 7f755effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861658299 0 0) 0x7f75700b05c0 con 0x7f7570072aa0 2024-03-25T18:25:47.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:25:47.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75700b09f0 con 0x7f7570072aa0 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 122454251 0 0) 0x7f7550004eb0 con 0x7f7570061830 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75700b05c0 con 0x7f7570061830 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7560001bc0 con 0x7f7570061830 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): dump: 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:47.068 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:47.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.066+0000 7f755effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:47.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2753860898 0 0) 0x7f75700b05c0 con 0x7f7570061830 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 >> v1:172.21.15.42:6790/0 conn(0x7f7570072aa0 legacy=0x7f75701cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 >> v1:172.21.15.120:6789/0 conn(0x7f757013c8c0 legacy=0x7f75701e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 1 monclient: found mon.a 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75701d8760 con 0x7f7570061830 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f755effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:47.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 10 monclient: _renew_subs 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75701d8950 con 0x7f7570061830 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 10 monclient: _renew_subs 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75701c7cb0 con 0x7f7570061830 2024-03-25T18:25:47.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.072+0000 7f75748ba640 1 librados: init done 2024-03-25T18:25:47.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.073+0000 7f75748ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.073+0000 7f75748ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.073+0000 7f75748ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:47.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.073+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75700b07b0 con 0x7f7570061830 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.400+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75600011f0 con 0x7f7570061830 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.400+0000 7f755effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.400+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7560004c30 con 0x7f7570061830 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.400+0000 7f755effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.400+0000 7f755effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.400+0000 7f755effd640 10 monclient: dump: 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.402 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.400+0000 7f756cd99640 4 set_mon_vals no callback set 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.401+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f756001ac80 con 0x7f7570061830 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.401+0000 7f756dd9b640 10 monclient: get_auth_request con 0x7f755006dcc0 auth_method 0 2024-03-25T18:25:47.403 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.402+0000 7f755effd640 10 client.4503.objecter ms_handle_connect 0x7f755006dcc0 2024-03-25T18:25:47.409 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f7560054970 con 0x7f7570061830 2024-03-25T18:25:47.409 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 10 client.4503.objecter ms_dispatch 0x7f7570000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:47.409 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:47.409 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:47.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T18:25:47.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75600502b0 con 0x7f7570061830 2024-03-25T18:25:47.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.408+0000 7f755effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:47.516 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:30.686 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.524 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:30.693 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.555 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:30.725 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.609 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.607+0000 7f75748ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.609 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.607+0000 7f75748ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.609 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.607+0000 7f75748ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:47.609 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.607+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75700627b0 con 0x7f7570061830 2024-03-25T18:25:47.663 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:30.784 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.669 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:30.790 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.697 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:30.866 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.701+0000 7f755effd640 1 -- 172.21.15.42:0/4025466498 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f75600523b0 con 0x7f7570061830 2024-03-25T18:25:47.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.701+0000 7f755effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.701+0000 7f755effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:47.703 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:47.703 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:47.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.703+0000 7f75748ba640 10 librados: watch_flush enter 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.703+0000 7f75748ba640 10 librados: watch_flush exit 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.703+0000 7f75748ba640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.703+0000 7f75748ba640 10 client.4503.objecter successfully canceled tick 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 >> v1:172.21.15.42:6800/3081750888 conn(0x7f755006dcc0 legacy=0x7f7550070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 10 monclient: shutdown 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 >> v1:172.21.15.42:6789/0 conn(0x7f7570061830 legacy=0x7f75701c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 shutdown_connections 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 >> 172.21.15.42:0/4025466498 conn(0x7f75700b0ef0 msgr2=0x7f757013fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 shutdown_connections 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 1 -- 172.21.15.42:0/4025466498 wait complete. 2024-03-25T18:25:47.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:47.704+0000 7f75748ba640 1 librados: shutdown 2024-03-25T18:25:47.707 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:30.827 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.712 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:30.881 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:47.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:47.742 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:30.912 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:54.523 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.640+0000 7fea71401640 10 monclient: get_monmap_and_config 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.640+0000 7fea71401640 10 monclient: build_initial_monmap 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.640+0000 7fea71401640 10 monclient: monmap: 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:25:54.641797+0000 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:25:54.641797+0000 2024-03-25T18:25:54.642 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:54.643 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:54.643 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:54.643 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:25:54.643 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:25:54.643 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:54.643 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.641+0000 7fea71401640 1 Processor -- start 2024-03-25T18:25:54.643 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.641+0000 7fea71401640 1 -- start start 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.641+0000 7fea71401640 10 monclient: init 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 10 monclient(hunting): picked mon.noname-b con 0x7fea6c0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 10 monclient(hunting): picked mon.noname-c con 0x7fea6c13f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 10 monclient(hunting): picked mon.noname-a con 0x7fea6c1430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c142520 con 0x7fea6c1430c0 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c0b28a0 con 0x7fea6c13f710 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c0b2a30 con 0x7fea6c0a98f0 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:54.644 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.642+0000 7fea71401640 10 monclient(hunting): authenticate will time out at 7669.869070s 2024-03-25T18:25:54.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.656+0000 7fea6a7fc640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fea6c13f710 0x7fea6c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:53074/0 (socket says 172.21.15.42:53074) 2024-03-25T18:25:54.657 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.656+0000 7fea6a7fc640 1 -- 172.21.15.42:0/2801955756 learned_addr learned my addr 172.21.15.42:0/2801955756 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:54.670 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.669+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037819856 0 0) 0x7fea6c0b28a0 con 0x7fea6c13f710 2024-03-25T18:25:54.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.669+0000 7fea69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.669+0000 7fea69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.669+0000 7fea69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.669+0000 7fea69ffb640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:25:54.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.670+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea54003520 con 0x7fea6c13f710 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2613312235 0 0) 0x7fea54003520 con 0x7fea6c13f710 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 >> v1:172.21.15.42:6790/0 conn(0x7fea6c0a98f0 legacy=0x7fea6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 >> v1:172.21.15.42:6789/0 conn(0x7fea6c1430c0 legacy=0x7fea6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 1 monclient: found mon.noname-c 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c0b44c0 con 0x7fea6c13f710 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:54.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fea60001690 con 0x7fea6c13f710 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea69ffb640 10 monclient: dump: 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea71401640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:25:54.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.681+0000 7fea71401640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea600017f0 con 0x7fea6c13f710 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea69ffb640 1 -- 172.21.15.42:0/2801955756 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fea60001c40 con 0x7fea6c13f710 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea71401640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea71401640 4 set_mon_vals no callback set 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.698+0000 7fea69ffb640 10 monclient: dump: 2024-03-25T18:25:54.699 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:54.700 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:54.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 10 monclient: shutdown 2024-03-25T18:25:54.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:54.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 1 -- 172.21.15.42:0/2801955756 >> v1:172.21.15.120:6789/0 conn(0x7fea6c13f710 legacy=0x7fea6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 1 -- 172.21.15.42:0/2801955756 shutdown_connections 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 1 -- 172.21.15.42:0/2801955756 >> 172.21.15.42:0/2801955756 conn(0x7fea6c0b7ea0 msgr2=0x7fea6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 1 -- 172.21.15.42:0/2801955756 shutdown_connections 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 1 -- 172.21.15.42:0/2801955756 wait complete. 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 10 monclient: build_initial_monmap 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.699+0000 7fea71401640 10 monclient: monmap: 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:25:54.702 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.700+0000 7fea71401640 1 librados: starting msgr at 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.700+0000 7fea71401640 1 librados: starting objecter 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.700+0000 7fea71401640 1 Processor -- start 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.700+0000 7fea71401640 1 -- start start 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.700+0000 7fea71401640 1 librados: setting wanted keys 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.700+0000 7fea71401640 1 librados: calling monclient init 2024-03-25T18:25:54.703 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.700+0000 7fea71401640 10 monclient: init 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 10 monclient(hunting): picked mon.noname-a con 0x7fea6c0a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 10 monclient(hunting): picked mon.noname-c con 0x7fea6c13f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 10 monclient(hunting): picked mon.noname-b con 0x7fea6c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c1370a0 con 0x7fea6c0a98f0 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c135090 con 0x7fea6c1430c0 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c135220 con 0x7fea6c13f710 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:54.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.701+0000 7fea71401640 10 monclient(hunting): authenticate will time out at 7669.927934s 2024-03-25T18:25:54.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.707+0000 7fea6a7fc640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fea6c13f710 0x7fea6c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:55008/0 (socket says 172.21.15.42:55008) 2024-03-25T18:25:54.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.707+0000 7fea6a7fc640 1 -- 172.21.15.42:0/595706097 learned_addr learned my addr 172.21.15.42:0/595706097 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:25:54.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.709+0000 7fea537fe640 10 client.?.objecter ms_handle_connect 0x7fea6c13f710 2024-03-25T18:25:54.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.709+0000 7fea537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.712+0000 7fea537fe640 10 client.?.objecter ms_handle_connect 0x7fea6c0a98f0 2024-03-25T18:25:54.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.712+0000 7fea537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.719+0000 7fea537fe640 10 client.?.objecter ms_handle_connect 0x7fea6c1430c0 2024-03-25T18:25:54.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.719+0000 7fea537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.725+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887335417 0 0) 0x7fea6c135220 con 0x7fea6c13f710 2024-03-25T18:25:54.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.725+0000 7fea537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.725+0000 7fea537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.725+0000 7fea537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.725+0000 7fea537fe640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:25:54.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.725+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea58004eb0 con 0x7fea6c13f710 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.727+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279573073 0 0) 0x7fea6c1370a0 con 0x7fea6c0a98f0 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.727+0000 7fea537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.727+0000 7fea537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.727+0000 7fea537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.727+0000 7fea537fe640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.727+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c135220 con 0x7fea6c0a98f0 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.746+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 867335743 0 0) 0x7fea6c135220 con 0x7fea6c0a98f0 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.746+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea6c1370a0 con 0x7fea6c0a98f0 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.746+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fea5c0018a0 con 0x7fea6c0a98f0 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.746+0000 7fea537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.746+0000 7fea537fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.746+0000 7fea537fe640 10 monclient(hunting): dump: 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.749 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:54.749 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:54.749 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:54.749 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:54.749 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:54.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.746+0000 7fea537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:54.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.747+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976699695 0 0) 0x7fea6c135090 con 0x7fea6c1430c0 2024-03-25T18:25:54.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.747+0000 7fea537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.747+0000 7fea537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.747+0000 7fea537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.747+0000 7fea537fe640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:25:54.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.747+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c135220 con 0x7fea6c1430c0 2024-03-25T18:25:54.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3150395573 0 0) 0x7fea6c135220 con 0x7fea6c1430c0 2024-03-25T18:25:54.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea6c135090 con 0x7fea6c1430c0 2024-03-25T18:25:54.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fea64001a90 con 0x7fea6c1430c0 2024-03-25T18:25:54.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:54.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 10 monclient(hunting): dump: 2024-03-25T18:25:54.761 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1196841514 0 0) 0x7fea6c1370a0 con 0x7fea6c0a98f0 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 >> v1:172.21.15.42:6790/0 conn(0x7fea6c13f710 legacy=0x7fea6c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 >> v1:172.21.15.120:6789/0 conn(0x7fea6c1430c0 legacy=0x7fea6c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 monclient: found mon.a 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c1d5000 con 0x7fea6c0a98f0 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea71401640 5 monclient: authenticate success, global_id 4509 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea71401640 10 monclient: _renew_subs 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea71401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.759+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c1d52b0 con 0x7fea6c0a98f0 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.760+0000 7fea71401640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.760+0000 7fea71401640 10 monclient: _renew_subs 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.760+0000 7fea71401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:54.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.760+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c135910 con 0x7fea6c0a98f0 2024-03-25T18:25:54.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.760+0000 7fea71401640 1 librados: init done 2024-03-25T18:25:54.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.761+0000 7fea71401640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:54.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.761+0000 7fea71401640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:54.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.761+0000 7fea71401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:54.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:54.761+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c0627b0 con 0x7fea6c0a98f0 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.071+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea5c001d50 con 0x7fea6c0a98f0 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.071+0000 7fea537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.071+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fea5c003790 con 0x7fea6c0a98f0 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.071+0000 7fea537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.071+0000 7fea537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.071+0000 7fea537fe640 10 monclient: dump: 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.073 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:25:55.074 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:25:55.074 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:25:55.074 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:25:55.074 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:25:55.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.071+0000 7fea697fa640 4 set_mon_vals no callback set 2024-03-25T18:25:55.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.072+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fea5c0039b0 con 0x7fea6c0a98f0 2024-03-25T18:25:55.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.073+0000 7fea6a7fc640 10 monclient: get_auth_request con 0x7fea5806dfd0 auth_method 0 2024-03-25T18:25:55.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.073+0000 7fea537fe640 10 client.4509.objecter ms_handle_connect 0x7fea5806dfd0 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.085+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fea5c053530 con 0x7fea6c0a98f0 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.085+0000 7fea537fe640 10 client.4509.objecter ms_dispatch 0x7fea6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.085+0000 7fea537fe640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.085+0000 7fea537fe640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.085+0000 7fea537fe640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.086+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea5c04ee70 con 0x7fea6c0a98f0 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.086+0000 7fea537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.086+0000 7fea537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:55.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.325+0000 7fea71401640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.325+0000 7fea71401640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.325+0000 7fea71401640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:25:55.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.325+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea6c13bfe0 con 0x7fea6c0a98f0 2024-03-25T18:25:55.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.365+0000 7fea537fe640 1 -- 172.21.15.42:0/595706097 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fea5c050f70 con 0x7fea6c0a98f0 2024-03-25T18:25:55.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.365+0000 7fea537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.365+0000 7fea537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:55.367 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:25:55.367 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 10 librados: watch_flush enter 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 10 librados: watch_flush exit 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 10 client.4509.objecter successfully canceled tick 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 >> v1:172.21.15.42:6800/3081750888 conn(0x7fea5806dfd0 legacy=0x7fea58070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 10 monclient: shutdown 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 >> v1:172.21.15.42:6789/0 conn(0x7fea6c0a98f0 legacy=0x7fea6c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 shutdown_connections 2024-03-25T18:25:55.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 >> 172.21.15.42:0/595706097 conn(0x7fea6c0b7ea0 msgr2=0x7fea6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:55.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 shutdown_connections 2024-03-25T18:25:55.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 1 -- 172.21.15.42:0/595706097 wait complete. 2024-03-25T18:25:55.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:25:55.368+0000 7fea71401640 1 librados: shutdown 2024-03-25T18:25:55.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:02.188 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:02.306 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.305+0000 7f4b408f6640 10 monclient: get_monmap_and_config 2024-03-25T18:26:02.306 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.305+0000 7f4b408f6640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.306 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.305+0000 7f4b408f6640 10 monclient: monmap: 2024-03-25T18:26:02.306 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:02.306 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:02.306290+0000 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:02.306290+0000 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:02.307 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:02.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 1 Processor -- start 2024-03-25T18:26:02.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 1 -- start start 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient: init 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient(hunting): picked mon.noname-a con 0x7f4b3c0b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3c0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient(hunting): picked mon.noname-b con 0x7f4b3c14be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0af940 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0b0a70 con 0x7f4b3c0b1730 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0b0d50 con 0x7f4b3c14be90 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.309 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.306+0000 7f4b408f6640 10 monclient(hunting): authenticate will time out at 7677.533345s 2024-03-25T18:26:02.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.322+0000 7f4b33fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f4b3c0b1730 0x7f4b3c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:53466/0 (socket says 172.21.15.42:53466) 2024-03-25T18:26:02.323 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.322+0000 7f4b33fff640 1 -- 172.21.15.42:0/2483292437 learned_addr learned my addr 172.21.15.42:0/2483292437 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.338+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776000498 0 0) 0x7f4b3c0b0a70 con 0x7f4b3c0b1730 2024-03-25T18:26:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.338+0000 7f4b39d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.338+0000 7f4b39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.338+0000 7f4b39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.338+0000 7f4b39d9b640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:26:02.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.338+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b28003520 con 0x7f4b3c0b1730 2024-03-25T18:26:02.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.346+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870100030 0 0) 0x7f4b3c0af940 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.346+0000 7f4b39d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.346+0000 7f4b39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.346+0000 7f4b39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.346+0000 7f4b39d9b640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:26:02.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.346+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c0b0a70 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.352+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184837806 0 0) 0x7f4b3c0b0d50 con 0x7f4b3c14be90 2024-03-25T18:26:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.352+0000 7f4b39d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.352+0000 7f4b39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.352+0000 7f4b39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.352+0000 7f4b39d9b640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:26:02.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.352+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c0af940 con 0x7f4b3c14be90 2024-03-25T18:26:02.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444603523 0 0) 0x7f4b3c0b0a70 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 >> v1:172.21.15.42:6790/0 conn(0x7f4b3c14be90 legacy=0x7f4b3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 >> v1:172.21.15.120:6789/0 conn(0x7f4b3c0b1730 legacy=0x7f4b3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 1 monclient: found mon.noname-a 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0b1030 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b2c001670 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b39d9b640 10 monclient: dump: 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:02.371 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b408f6640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:26:02.372 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.369+0000 7f4b408f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:02.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b2c0017d0 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b39d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:02.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b39d9b640 1 -- 172.21.15.42:0/2483292437 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b2c001be0 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b39d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b39d9b640 10 monclient: dump: 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b408f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:02.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.384+0000 7f4b408f6640 4 set_mon_vals no callback set 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.385+0000 7f4b408f6640 10 monclient: shutdown 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.385+0000 7f4b408f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.385+0000 7f4b408f6640 1 -- 172.21.15.42:0/2483292437 >> v1:172.21.15.42:6789/0 conn(0x7f4b3c0b4d60 legacy=0x7f4b3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.385+0000 7f4b408f6640 1 -- 172.21.15.42:0/2483292437 shutdown_connections 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.385+0000 7f4b408f6640 1 -- 172.21.15.42:0/2483292437 >> 172.21.15.42:0/2483292437 conn(0x7f4b3c0617a0 msgr2=0x7f4b3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.385+0000 7f4b408f6640 1 -- 172.21.15.42:0/2483292437 shutdown_connections 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.385+0000 7f4b408f6640 1 -- 172.21.15.42:0/2483292437 wait complete. 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.386+0000 7f4b408f6640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.386+0000 7f4b408f6640 10 monclient: monmap: 2024-03-25T18:26:02.388 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 1 librados: starting msgr at 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 1 librados: starting objecter 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 1 Processor -- start 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 1 -- start start 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 1 librados: setting wanted keys 2024-03-25T18:26:02.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 1 librados: calling monclient init 2024-03-25T18:26:02.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 10 monclient: init 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3c0b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 10 monclient(hunting): picked mon.noname-a con 0x7f4b3c0b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.387+0000 7f4b408f6640 10 monclient(hunting): picked mon.noname-b con 0x7f4b3c14be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.388+0000 7f4b408f6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c139480 con 0x7f4b3c0b4d60 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.388+0000 7f4b408f6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c134a90 con 0x7f4b3c14be90 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.388+0000 7f4b408f6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c13a830 con 0x7f4b3c0b1730 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.388+0000 7f4b408f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.388+0000 7f4b408f6640 10 monclient(hunting): authenticate will time out at 7677.614621s 2024-03-25T18:26:02.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.395+0000 7f4b3ad9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f4b3c14be90 0x7f4b3c134380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:53506/0 (socket says 172.21.15.42:53506) 2024-03-25T18:26:02.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.395+0000 7f4b3ad9d640 1 -- 172.21.15.42:0/3548106224 learned_addr learned my addr 172.21.15.42:0/3548106224 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.396+0000 7f4b32ffd640 10 client.?.objecter ms_handle_connect 0x7f4b3c14be90 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.396+0000 7f4b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.403+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421920059 0 0) 0x7f4b3c134a90 con 0x7f4b3c14be90 2024-03-25T18:26:02.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.403+0000 7f4b32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.403+0000 7f4b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.403+0000 7f4b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.403+0000 7f4b32ffd640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:26:02.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.403+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b1c004eb0 con 0x7f4b3c14be90 2024-03-25T18:26:02.411 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.410+0000 7f4b32ffd640 10 client.?.objecter ms_handle_connect 0x7f4b3c0b1730 2024-03-25T18:26:02.411 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.410+0000 7f4b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.412+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596634927 0 0) 0x7f4b1c004eb0 con 0x7f4b3c14be90 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.412+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b3c134a90 con 0x7f4b3c14be90 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.412+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b34001c70 con 0x7f4b3c14be90 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.412+0000 7f4b32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.412+0000 7f4b32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.412+0000 7f4b32ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:02.413 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:02.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.412+0000 7f4b32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:02.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2992936018 0 0) 0x7f4b3c134a90 con 0x7f4b3c14be90 2024-03-25T18:26:02.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 >> v1:172.21.15.42:6790/0 conn(0x7f4b3c0b1730 legacy=0x7f4b3c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 >> v1:172.21.15.42:6789/0 conn(0x7f4b3c0b4d60 legacy=0x7f4b3c133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 1 monclient: found mon.b 2024-03-25T18:26:02.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c13ab10 con 0x7f4b3c14be90 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 10 monclient: _renew_subs 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c143310 con 0x7f4b3c14be90 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 10 monclient: _renew_subs 2024-03-25T18:26:02.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c138f60 con 0x7f4b3c14be90 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.417+0000 7f4b408f6640 1 librados: init done 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.418+0000 7f4b408f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.418+0000 7f4b408f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.418+0000 7f4b408f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:02.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.418+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c1401f0 con 0x7f4b3c14be90 2024-03-25T18:26:02.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.638+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b34003990 con 0x7f4b3c14be90 2024-03-25T18:26:02.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.638+0000 7f4b32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:02.640 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.638+0000 7f4b3959a640 4 set_mon_vals no callback set 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.689+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f4b34003da0 con 0x7f4b3c14be90 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.689+0000 7f4b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.689+0000 7f4b32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.689+0000 7f4b32ffd640 10 monclient: dump: 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:02.692 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:02.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.690+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f4b34003f80 con 0x7f4b3c14be90 2024-03-25T18:26:02.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.691+0000 7f4b3a59c640 10 monclient: get_auth_request con 0x7f4b1c06dc20 auth_method 0 2024-03-25T18:26:02.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.691+0000 7f4b32ffd640 10 client.4519.objecter ms_handle_connect 0x7f4b1c06dc20 2024-03-25T18:26:02.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.693+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f4b34054b10 con 0x7f4b3c14be90 2024-03-25T18:26:02.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.693+0000 7f4b32ffd640 10 client.4519.objecter ms_dispatch 0x7f4b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:02.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.693+0000 7f4b32ffd640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:02.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.693+0000 7f4b32ffd640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:02.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.693+0000 7f4b32ffd640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-25T18:26:02.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.695+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b34050620 con 0x7f4b3c14be90 2024-03-25T18:26:02.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.695+0000 7f4b32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.695+0000 7f4b32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:02.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.914+0000 7f4b408f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:02.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.914+0000 7f4b408f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:02.916 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.914+0000 7f4b408f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:02.916 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.914+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b3c139210 con 0x7f4b3c14be90 2024-03-25T18:26:02.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.945+0000 7f4b32ffd640 1 -- 172.21.15.42:0/3548106224 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f4b34052550 con 0x7f4b3c14be90 2024-03-25T18:26:02.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.945+0000 7f4b32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:02.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.945+0000 7f4b32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:02.947 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:02.947 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 10 librados: watch_flush enter 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 10 librados: watch_flush exit 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 10 client.4519.objecter successfully canceled tick 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 >> v1:172.21.15.42:6800/3081750888 conn(0x7f4b1c06dc20 legacy=0x7f4b1c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 10 monclient: shutdown 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 >> v1:172.21.15.120:6789/0 conn(0x7f4b3c14be90 legacy=0x7f4b3c134380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 shutdown_connections 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 >> 172.21.15.42:0/3548106224 conn(0x7f4b3c0617a0 msgr2=0x7f4b3c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:02.950 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 shutdown_connections 2024-03-25T18:26:02.951 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 1 -- 172.21.15.42:0/3548106224 wait complete. 2024-03-25T18:26:02.951 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:02.948+0000 7f4b408f6640 1 librados: shutdown 2024-03-25T18:26:02.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:07.540 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:50.709 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.546 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:50.715 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.585 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:06:50.754 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.604 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:50.773 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.609 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:50.779 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.619 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:50.739 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.623 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:50.744 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.642 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:06:50.812 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:07.654 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:06:50.774 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:09.767 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:09.885 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.884+0000 7fb2d9d6d640 10 monclient: get_monmap_and_config 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.884+0000 7fb2d9d6d640 10 monclient: build_initial_monmap 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.884+0000 7fb2d9d6d640 10 monclient: monmap: 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:09.885739+0000 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:09.885739+0000 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:09.886 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:09.887 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.885+0000 7fb2d9d6d640 1 Processor -- start 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.885+0000 7fb2d9d6d640 1 -- start start 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.885+0000 7fb2d9d6d640 10 monclient: init 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d40a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d41430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4142520 con 0x7fb2d413f710 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40b28a0 con 0x7fb2d40a98f0 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40b2a30 con 0x7fb2d41430c0 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:09.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.886+0000 7fb2d9d6d640 10 monclient(hunting): authenticate will time out at 7685.112959s 2024-03-25T18:26:09.899 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.898+0000 7fb2d3fff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fb2d41430c0 0x7fb2d414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:47298/0 (socket says 172.21.15.42:47298) 2024-03-25T18:26:09.899 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.898+0000 7fb2d3fff640 1 -- 172.21.15.42:0/3235357858 learned_addr learned my addr 172.21.15.42:0/3235357858 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:09.912 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.911+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411697125 0 0) 0x7fb2d40b2a30 con 0x7fb2d41430c0 2024-03-25T18:26:09.912 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.911+0000 7fb2d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.911+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.911+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.911+0000 7fb2d2ffd640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:26:09.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.911+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0003520 con 0x7fb2d41430c0 2024-03-25T18:26:09.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.916+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960261228 0 0) 0x7fb2d4142520 con 0x7fb2d413f710 2024-03-25T18:26:09.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.916+0000 7fb2d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.916+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.916+0000 7fb2d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.916+0000 7fb2d2ffd640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:26:09.918 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.916+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d40b2a30 con 0x7fb2d413f710 2024-03-25T18:26:09.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.923+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691299776 0 0) 0x7fb2c0003520 con 0x7fb2d41430c0 2024-03-25T18:26:09.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.923+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 >> v1:172.21.15.120:6789/0 conn(0x7fb2d40a98f0 legacy=0x7fb2d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.923+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 >> v1:172.21.15.42:6789/0 conn(0x7fb2d413f710 legacy=0x7fb2d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.923+0000 7fb2d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:09.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.923+0000 7fb2d2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:26:09.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.923+0000 7fb2d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:09.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d40b44c0 con 0x7fb2d41430c0 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb2cc001a10 con 0x7fb2d41430c0 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d2ffd640 10 monclient: dump: 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.926 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:09.927 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:09.927 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:09.927 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:09.927 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:09.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d9d6d640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:26:09.927 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.924+0000 7fb2d9d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2cc001b70 con 0x7fb2d41430c0 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d2ffd640 1 -- 172.21.15.42:0/3235357858 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb2cc001f80 con 0x7fb2d41430c0 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d2ffd640 10 monclient: dump: 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:09.935 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:09.936 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:09.936 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:09.936 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:09.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d9d6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:09.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.933+0000 7fb2d9d6d640 4 set_mon_vals no callback set 2024-03-25T18:26:09.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.934+0000 7fb2d9d6d640 10 monclient: shutdown 2024-03-25T18:26:09.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.934+0000 7fb2d9d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:09.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.934+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3235357858 >> v1:172.21.15.42:6790/0 conn(0x7fb2d41430c0 legacy=0x7fb2d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.934+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3235357858 shutdown_connections 2024-03-25T18:26:09.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.934+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3235357858 >> 172.21.15.42:0/3235357858 conn(0x7fb2d40b7ea0 msgr2=0x7fb2d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.934+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3235357858 shutdown_connections 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.934+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3235357858 wait complete. 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.935+0000 7fb2d9d6d640 10 monclient: build_initial_monmap 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.935+0000 7fb2d9d6d640 10 monclient: monmap: 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:09.939 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.935+0000 7fb2d9d6d640 1 librados: starting msgr at 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.935+0000 7fb2d9d6d640 1 librados: starting objecter 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 1 Processor -- start 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 1 -- start start 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 1 librados: setting wanted keys 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 1 librados: calling monclient init 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 10 monclient: init 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.936+0000 7fb2d9d6d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d41430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.937+0000 7fb2d9d6d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d413c9b0 con 0x7fb2d413f710 2024-03-25T18:26:09.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.937+0000 7fb2d9d6d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d41397c0 con 0x7fb2d41430c0 2024-03-25T18:26:09.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.937+0000 7fb2d9d6d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4139950 con 0x7fb2d40a98f0 2024-03-25T18:26:09.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.937+0000 7fb2d9d6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:09.941 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.937+0000 7fb2d9d6d640 10 monclient(hunting): authenticate will time out at 7685.163739s 2024-03-25T18:26:09.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.947+0000 7fb2d3fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb2d41430c0 0x7fb2d41e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34062/0 (socket says 172.21.15.42:34062) 2024-03-25T18:26:09.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.947+0000 7fb2d3fff640 1 -- 172.21.15.42:0/3528441690 learned_addr learned my addr 172.21.15.42:0/3528441690 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:09.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.947+0000 7fb2d0ff9640 10 client.?.objecter ms_handle_connect 0x7fb2d41430c0 2024-03-25T18:26:09.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.947+0000 7fb2d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:09.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451050247 0 0) 0x7fb2d41397c0 con 0x7fb2d41430c0 2024-03-25T18:26:09.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:26:09.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b4004eb0 con 0x7fb2d41430c0 2024-03-25T18:26:09.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 10 client.?.objecter ms_handle_connect 0x7fb2d413f710 2024-03-25T18:26:09.962 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.960+0000 7fb2d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:09.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.969+0000 7fb2d0ff9640 10 client.?.objecter ms_handle_connect 0x7fb2d40a98f0 2024-03-25T18:26:09.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.969+0000 7fb2d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:09.981 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.980+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116802202 0 0) 0x7fb2d413c9b0 con 0x7fb2d413f710 2024-03-25T18:26:09.982 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.980+0000 7fb2d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.982 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.980+0000 7fb2d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.982 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.980+0000 7fb2d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.982 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.980+0000 7fb2d0ff9640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:26:09.982 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.980+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d41397c0 con 0x7fb2d413f710 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.986+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590136806 0 0) 0x7fb2b4004eb0 con 0x7fb2d41430c0 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.986+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2d413c9b0 con 0x7fb2d41430c0 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.986+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb2cc001ec0 con 0x7fb2d41430c0 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.986+0000 7fb2d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.986+0000 7fb2d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.986+0000 7fb2d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:09.987 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:09.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.986+0000 7fb2d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:09.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.992+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296294676 0 0) 0x7fb2d4139950 con 0x7fb2d40a98f0 2024-03-25T18:26:09.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.992+0000 7fb2d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.992+0000 7fb2d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.993 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.992+0000 7fb2d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.992+0000 7fb2d0ff9640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:26:09.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:09.992+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b4004eb0 con 0x7fb2d40a98f0 2024-03-25T18:26:10.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3212680867 0 0) 0x7fb2d413c9b0 con 0x7fb2d41430c0 2024-03-25T18:26:10.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 >> v1:172.21.15.42:6790/0 conn(0x7fb2d40a98f0 legacy=0x7fb2d4130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 >> v1:172.21.15.42:6789/0 conn(0x7fb2d413f710 legacy=0x7fb2d41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.003 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 1 monclient: found mon.b 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4131680 con 0x7fb2d41430c0 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 10 monclient: _renew_subs 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d4131950 con 0x7fb2d41430c0 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:10.004 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 10 monclient: _renew_subs 2024-03-25T18:26:10.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:10.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d4135a30 con 0x7fb2d41430c0 2024-03-25T18:26:10.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.002+0000 7fb2d9d6d640 1 librados: init done 2024-03-25T18:26:10.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.003+0000 7fb2d9d6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.003+0000 7fb2d9d6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.003+0000 7fb2d9d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:10.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.003+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d413c810 con 0x7fb2d41430c0 2024-03-25T18:26:10.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.296+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2cc004e10 con 0x7fb2d41430c0 2024-03-25T18:26:10.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.296+0000 7fb2d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:10.298 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.296+0000 7fb2d27fc640 4 set_mon_vals no callback set 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.297+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb2cc002100 con 0x7fb2d41430c0 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.297+0000 7fb2d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.297+0000 7fb2d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.297+0000 7fb2d0ff9640 10 monclient: dump: 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:10.299 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:10.300 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:10.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.297+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fb2cc01adf0 con 0x7fb2d41430c0 2024-03-25T18:26:10.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.298+0000 7fb2d37fe640 10 monclient: get_auth_request con 0x7fb2b406de00 auth_method 0 2024-03-25T18:26:10.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.298+0000 7fb2d0ff9640 10 client.4525.objecter ms_handle_connect 0x7fb2b406de00 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fb2cc054930 con 0x7fb2d41430c0 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 10 client.4525.objecter ms_dispatch 0x7fb2d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:10.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:10.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:10.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T18:26:10.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2cc050440 con 0x7fb2d41430c0 2024-03-25T18:26:10.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.302+0000 7fb2d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:10.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.558+0000 7fb2d9d6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.558+0000 7fb2d9d6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.558+0000 7fb2d9d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:10.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.558+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2d4136fc0 con 0x7fb2d41430c0 2024-03-25T18:26:10.596 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.595+0000 7fb2d0ff9640 1 -- 172.21.15.42:0/3528441690 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fb2cc052370 con 0x7fb2d41430c0 2024-03-25T18:26:10.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.595+0000 7fb2d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.595+0000 7fb2d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 10 librados: watch_flush enter 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 10 librados: watch_flush exit 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 10 client.4525.objecter successfully canceled tick 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 >> v1:172.21.15.42:6800/3081750888 conn(0x7fb2b406de00 legacy=0x7fb2b40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 10 monclient: shutdown 2024-03-25T18:26:10.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:10.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.597+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 >> v1:172.21.15.120:6789/0 conn(0x7fb2d41430c0 legacy=0x7fb2d41e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.598+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 shutdown_connections 2024-03-25T18:26:10.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.598+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 >> 172.21.15.42:0/3528441690 conn(0x7fb2d40b7ea0 msgr2=0x7fb2d414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:10.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.598+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 shutdown_connections 2024-03-25T18:26:10.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.598+0000 7fb2d9d6d640 1 -- 172.21.15.42:0/3528441690 wait complete. 2024-03-25T18:26:10.625 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:10.598+0000 7fb2d9d6d640 1 librados: shutdown 2024-03-25T18:26:10.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:17.433 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:17.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.548+0000 7f6e89884640 10 monclient: get_monmap_and_config 2024-03-25T18:26:17.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.548+0000 7f6e89884640 10 monclient: build_initial_monmap 2024-03-25T18:26:17.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.549+0000 7f6e89884640 10 monclient: monmap: 2024-03-25T18:26:17.550 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:17.550 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:17.550 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:17.550039+0000 2024-03-25T18:26:17.551 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:17.550039+0000 2024-03-25T18:26:17.551 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:17.551 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:17.551 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:17.551 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:17.551 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:17.551 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:17.552 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.549+0000 7f6e89884640 1 Processor -- start 2024-03-25T18:26:17.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.549+0000 7f6e89884640 1 -- start start 2024-03-25T18:26:17.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.549+0000 7f6e89884640 10 monclient: init 2024-03-25T18:26:17.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:17.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:17.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 10 monclient(hunting): picked mon.noname-a con 0x7f6e840a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:17.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:17.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 10 monclient(hunting): picked mon.noname-b con 0x7f6e841430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:17.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e84142520 con 0x7f6e840a98f0 2024-03-25T18:26:17.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840b28a0 con 0x7f6e8413f710 2024-03-25T18:26:17.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840b2a30 con 0x7f6e841430c0 2024-03-25T18:26:17.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:17.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.550+0000 7f6e89884640 10 monclient(hunting): authenticate will time out at 7692.777333s 2024-03-25T18:26:17.575 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.573+0000 7f6e827fc640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f6e8413f710 0x7f6e841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:59120/0 (socket says 172.21.15.42:59120) 2024-03-25T18:26:17.575 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.573+0000 7f6e827fc640 1 -- 172.21.15.42:0/2380188802 learned_addr learned my addr 172.21.15.42:0/2380188802 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:17.593 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.592+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708023916 0 0) 0x7f6e840b28a0 con 0x7f6e8413f710 2024-03-25T18:26:17.593 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.592+0000 7f6e81ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.593 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.592+0000 7f6e81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.593 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.592+0000 7f6e81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.594 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.592+0000 7f6e81ffb640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:26:17.594 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.592+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6c003520 con 0x7f6e8413f710 2024-03-25T18:26:17.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374800410 0 0) 0x7f6e6c003520 con 0x7f6e8413f710 2024-03-25T18:26:17.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 >> v1:172.21.15.42:6790/0 conn(0x7f6e841430c0 legacy=0x7f6e8414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 >> v1:172.21.15.42:6789/0 conn(0x7f6e840a98f0 legacy=0x7f6e840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:17.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 1 monclient: found mon.noname-c 2024-03-25T18:26:17.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:17.605 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840b44c0 con 0x7f6e8413f710 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6e78001690 con 0x7f6e8413f710 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e81ffb640 10 monclient: dump: 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.606 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:17.607 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:17.607 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:17.607 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:17.607 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:17.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e89884640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:26:17.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.604+0000 7f6e89884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:17.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e780017f0 con 0x7f6e8413f710 2024-03-25T18:26:17.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:17.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e81ffb640 1 -- 172.21.15.42:0/2380188802 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6e78001c00 con 0x7f6e8413f710 2024-03-25T18:26:17.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:17.617 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e81ffb640 10 monclient: dump: 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e89884640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:17.618 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.616+0000 7f6e89884640 4 set_mon_vals no callback set 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.617+0000 7f6e89884640 10 monclient: shutdown 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.617+0000 7f6e89884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.617+0000 7f6e89884640 1 -- 172.21.15.42:0/2380188802 >> v1:172.21.15.120:6789/0 conn(0x7f6e8413f710 legacy=0x7f6e841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.617+0000 7f6e89884640 1 -- 172.21.15.42:0/2380188802 shutdown_connections 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.617+0000 7f6e89884640 1 -- 172.21.15.42:0/2380188802 >> 172.21.15.42:0/2380188802 conn(0x7f6e840b7ea0 msgr2=0x7f6e840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.617+0000 7f6e89884640 1 -- 172.21.15.42:0/2380188802 shutdown_connections 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.617+0000 7f6e89884640 1 -- 172.21.15.42:0/2380188802 wait complete. 2024-03-25T18:26:17.620 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.618+0000 7f6e89884640 10 monclient: build_initial_monmap 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.618+0000 7f6e89884640 10 monclient: monmap: 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.618+0000 7f6e89884640 1 librados: starting msgr at 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.618+0000 7f6e89884640 1 librados: starting objecter 2024-03-25T18:26:17.621 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.619+0000 7f6e89884640 1 Processor -- start 2024-03-25T18:26:17.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.619+0000 7f6e89884640 1 -- start start 2024-03-25T18:26:17.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.619+0000 7f6e89884640 1 librados: setting wanted keys 2024-03-25T18:26:17.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.619+0000 7f6e89884640 1 librados: calling monclient init 2024-03-25T18:26:17.622 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.619+0000 7f6e89884640 10 monclient: init 2024-03-25T18:26:17.623 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:17.623 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:17.623 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 10 monclient(hunting): picked mon.noname-c con 0x7f6e840a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:17.623 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 10 monclient(hunting): picked mon.noname-b con 0x7f6e8413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:17.623 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 10 monclient(hunting): picked mon.noname-a con 0x7f6e841430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:17.623 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840abdc0 con 0x7f6e841430c0 2024-03-25T18:26:17.623 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e841d7730 con 0x7f6e8413f710 2024-03-25T18:26:17.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e84138f10 con 0x7f6e840a98f0 2024-03-25T18:26:17.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:17.624 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.620+0000 7f6e89884640 10 monclient(hunting): authenticate will time out at 7692.846987s 2024-03-25T18:26:17.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.629+0000 7f6e82ffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f6e840a98f0 0x7f6e84138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:36616/0 (socket says 172.21.15.42:36616) 2024-03-25T18:26:17.630 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.629+0000 7f6e82ffd640 1 -- 172.21.15.42:0/2152719794 learned_addr learned my addr 172.21.15.42:0/2152719794 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:17.634 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.633+0000 7f6e6bfff640 10 client.?.objecter ms_handle_connect 0x7f6e8413f710 2024-03-25T18:26:17.634 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.633+0000 7f6e6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.636 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.635+0000 7f6e6bfff640 10 client.?.objecter ms_handle_connect 0x7f6e840a98f0 2024-03-25T18:26:17.636 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.635+0000 7f6e6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.645 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.644+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852392598 0 0) 0x7f6e841d7730 con 0x7f6e8413f710 2024-03-25T18:26:17.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.644+0000 7f6e6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.644+0000 7f6e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.644+0000 7f6e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.644+0000 7f6e6bfff640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:26:17.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.644+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70004eb0 con 0x7f6e8413f710 2024-03-25T18:26:17.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.647+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481484043 0 0) 0x7f6e84138f10 con 0x7f6e840a98f0 2024-03-25T18:26:17.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.647+0000 7f6e6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.647+0000 7f6e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.647+0000 7f6e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.647+0000 7f6e6bfff640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:26:17.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.647+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e841d7730 con 0x7f6e840a98f0 2024-03-25T18:26:17.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.658+0000 7f6e6bfff640 10 client.?.objecter ms_handle_connect 0x7f6e841430c0 2024-03-25T18:26:17.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.658+0000 7f6e6bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.667+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2443171653 0 0) 0x7f6e70004eb0 con 0x7f6e8413f710 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.667+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e84138f10 con 0x7f6e8413f710 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.667+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6e78001bc0 con 0x7f6e8413f710 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.667+0000 7f6e6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.668+0000 7f6e6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.668+0000 7f6e6bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.669 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:17.670 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:17.670 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:17.670 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:17.670 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:17.670 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.668+0000 7f6e6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:17.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.674+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093339794 0 0) 0x7f6e840abdc0 con 0x7f6e841430c0 2024-03-25T18:26:17.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.674+0000 7f6e6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.674+0000 7f6e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.674+0000 7f6e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.674+0000 7f6e6bfff640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:26:17.676 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.674+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70004eb0 con 0x7f6e841430c0 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1285983221 0 0) 0x7f6e84138f10 con 0x7f6e8413f710 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 >> v1:172.21.15.42:6790/0 conn(0x7f6e840a98f0 legacy=0x7f6e84138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 >> v1:172.21.15.42:6789/0 conn(0x7f6e841430c0 legacy=0x7f6e841d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 1 monclient: found mon.b 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e841390a0 con 0x7f6e8413f710 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:26:17.684 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 10 monclient: _renew_subs 2024-03-25T18:26:17.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:17.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e841d8320 con 0x7f6e8413f710 2024-03-25T18:26:17.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:17.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 10 monclient: _renew_subs 2024-03-25T18:26:17.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:17.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e841d43c0 con 0x7f6e8413f710 2024-03-25T18:26:17.685 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.682+0000 7f6e89884640 1 librados: init done 2024-03-25T18:26:17.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.683+0000 7f6e89884640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.683+0000 7f6e89884640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.683+0000 7f6e89884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:17.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.683+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e840627b0 con 0x7f6e8413f710 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.894+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e78004a90 con 0x7f6e8413f710 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.894+0000 7f6e6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.894+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6e78004da0 con 0x7f6e8413f710 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.894+0000 7f6e6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.894+0000 7f6e6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.894+0000 7f6e6bfff640 10 monclient: dump: 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:17.896 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.894+0000 7f6e817fa640 4 set_mon_vals no callback set 2024-03-25T18:26:17.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.950+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f6e7801ad70 con 0x7f6e8413f710 2024-03-25T18:26:17.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.951+0000 7f6e82ffd640 10 monclient: get_auth_request con 0x7f6e7006de00 auth_method 0 2024-03-25T18:26:17.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.951+0000 7f6e6bfff640 10 client.4531.objecter ms_handle_connect 0x7f6e7006de00 2024-03-25T18:26:17.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.956+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f6e78054900 con 0x7f6e8413f710 2024-03-25T18:26:17.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.956+0000 7f6e6bfff640 10 client.4531.objecter ms_dispatch 0x7f6e84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:17.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.956+0000 7f6e6bfff640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:17.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.956+0000 7f6e6bfff640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:17.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.956+0000 7f6e6bfff640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.958+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e78050410 con 0x7f6e8413f710 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.958+0000 7f6e6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:17.958+0000 7f6e6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:18.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.184+0000 7f6e89884640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.184+0000 7f6e89884640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.184+0000 7f6e89884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:18.186 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.184+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e84069f40 con 0x7f6e8413f710 2024-03-25T18:26:18.249 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.248+0000 7f6e6bfff640 1 -- 172.21.15.42:0/2152719794 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f6e78052340 con 0x7f6e8413f710 2024-03-25T18:26:18.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.248+0000 7f6e6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.248+0000 7f6e6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:18.250 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:18.250 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 10 librados: watch_flush enter 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 10 librados: watch_flush exit 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 10 client.4531.objecter successfully canceled tick 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 >> v1:172.21.15.42:6800/3081750888 conn(0x7f6e7006de00 legacy=0x7f6e700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 10 monclient: shutdown 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:18.252 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 >> v1:172.21.15.120:6789/0 conn(0x7f6e8413f710 legacy=0x7f6e841d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 shutdown_connections 2024-03-25T18:26:18.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 >> 172.21.15.42:0/2152719794 conn(0x7f6e840b7ea0 msgr2=0x7f6e84074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:18.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 shutdown_connections 2024-03-25T18:26:18.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 1 -- 172.21.15.42:0/2152719794 wait complete. 2024-03-25T18:26:18.253 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:18.250+0000 7f6e89884640 1 librados: shutdown 2024-03-25T18:26:18.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:25.073 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:25.191 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.190+0000 7fa5d8739640 10 monclient: get_monmap_and_config 2024-03-25T18:26:25.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.190+0000 7fa5d8739640 10 monclient: build_initial_monmap 2024-03-25T18:26:25.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.190+0000 7fa5d8739640 10 monclient: monmap: 2024-03-25T18:26:25.192 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:25.192 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:25.192 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:25.191372+0000 2024-03-25T18:26:25.192 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:25.191372+0000 2024-03-25T18:26:25.192 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:25.193 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:25.193 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:25.193 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:25.193 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:25.193 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:25.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.191+0000 7fa5d8739640 1 Processor -- start 2024-03-25T18:26:25.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.191+0000 7fa5d8739640 1 -- start start 2024-03-25T18:26:25.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.191+0000 7fa5d8739640 10 monclient: init 2024-03-25T18:26:25.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:25.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:25.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 10 monclient(hunting): picked mon.noname-a con 0x7fa5d00ae6e0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:25.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 10 monclient(hunting): picked mon.noname-b con 0x7fa5d006f080 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:25.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 10 monclient(hunting): picked mon.noname-c con 0x7fa5d00704e0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:25.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d00718d0 con 0x7fa5d00ae6e0 2024-03-25T18:26:25.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d006fa30 con 0x7fa5d00704e0 2024-03-25T18:26:25.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d0070090 con 0x7fa5d006f080 2024-03-25T18:26:25.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:25.196 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.192+0000 7fa5d8739640 10 monclient(hunting): authenticate will time out at 7700.418900s 2024-03-25T18:26:25.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.200+0000 7fa5d6cd6640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa5d00704e0 0x7fa5d014a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:35520/0 (socket says 172.21.15.42:35520) 2024-03-25T18:26:25.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.200+0000 7fa5d6cd6640 1 -- 172.21.15.42:0/1847204893 learned_addr learned my addr 172.21.15.42:0/1847204893 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:25.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.209+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753736138 0 0) 0x7fa5d006fa30 con 0x7fa5d00704e0 2024-03-25T18:26:25.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.209+0000 7fa5d54d3640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.209+0000 7fa5d54d3640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.209+0000 7fa5d54d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.209+0000 7fa5d54d3640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:26:25.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.209+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5bc003520 con 0x7fa5d00704e0 2024-03-25T18:26:25.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631561094 0 0) 0x7fa5bc003520 con 0x7fa5d00704e0 2024-03-25T18:26:25.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 >> v1:172.21.15.42:6790/0 conn(0x7fa5d006f080 legacy=0x7fa5d0071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 >> v1:172.21.15.42:6789/0 conn(0x7fa5d00ae6e0 legacy=0x7fa5d013c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 10 monclient: _finish_hunting 0 2024-03-25T18:26:25.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 1 monclient: found mon.noname-c 2024-03-25T18:26:25.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:25.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:26:25.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d006f510 con 0x7fa5d00704e0 2024-03-25T18:26:25.221 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 10 monclient: _finish_auth 0 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 10 monclient: _check_auth_tickets 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa5c8001a10 con 0x7fa5d00704e0 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d54d3640 10 monclient: dump: 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.222 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.223 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:25.223 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:25.223 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:25.223 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:25.223 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:25.223 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d8739640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:26:25.223 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.219+0000 7fa5d8739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:25.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5c8001b70 con 0x7fa5d00704e0 2024-03-25T18:26:25.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d54d3640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:25.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d54d3640 1 -- 172.21.15.42:0/1847204893 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa5c8001f80 con 0x7fa5d00704e0 2024-03-25T18:26:25.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d8739640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:25.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d8739640 4 set_mon_vals no callback set 2024-03-25T18:26:25.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d54d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.225 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d54d3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.223+0000 7fa5d54d3640 10 monclient: dump: 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:25.226 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.224+0000 7fa5d8739640 10 monclient: shutdown 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.224+0000 7fa5d8739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.224+0000 7fa5d8739640 1 -- 172.21.15.42:0/1847204893 >> v1:172.21.15.120:6789/0 conn(0x7fa5d00704e0 legacy=0x7fa5d014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.224+0000 7fa5d8739640 1 -- 172.21.15.42:0/1847204893 shutdown_connections 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.224+0000 7fa5d8739640 1 -- 172.21.15.42:0/1847204893 >> 172.21.15.42:0/1847204893 conn(0x7fa5d006b630 msgr2=0x7fa5d00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.225+0000 7fa5d8739640 1 -- 172.21.15.42:0/1847204893 shutdown_connections 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.225+0000 7fa5d8739640 1 -- 172.21.15.42:0/1847204893 wait complete. 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 10 monclient: build_initial_monmap 2024-03-25T18:26:25.229 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 10 monclient: monmap: 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 1 librados: starting msgr at 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 1 librados: starting objecter 2024-03-25T18:26:25.230 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 1 Processor -- start 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 1 -- start start 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 1 librados: setting wanted keys 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 1 librados: calling monclient init 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.226+0000 7fa5d8739640 10 monclient: init 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 10 monclient(hunting): picked mon.noname-b con 0x7fa5d00704e0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 10 monclient(hunting): picked mon.noname-a con 0x7fa5d00ae6e0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 10 monclient(hunting): picked mon.noname-c con 0x7fa5d011bcc0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d011d2f0 con 0x7fa5d00ae6e0 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d011c180 con 0x7fa5d00704e0 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d013eab0 con 0x7fa5d011bcc0 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:25.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.227+0000 7fa5d8739640 10 monclient(hunting): authenticate will time out at 7700.454228s 2024-03-25T18:26:25.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.248+0000 7fa5d64d5640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa5d00704e0 0x7fa5d0136eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:35562/0 (socket says 172.21.15.42:35562) 2024-03-25T18:26:25.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.248+0000 7fa5d64d5640 1 -- 172.21.15.42:0/3917646342 learned_addr learned my addr 172.21.15.42:0/3917646342 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:25.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.250+0000 7fa5c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa5d00704e0 2024-03-25T18:26:25.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.250+0000 7fa5c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.256 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.255+0000 7fa5c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa5d00ae6e0 2024-03-25T18:26:25.256 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.255+0000 7fa5c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.256 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.255+0000 7fa5c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa5d011bcc0 2024-03-25T18:26:25.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.255+0000 7fa5c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.263+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056004595 0 0) 0x7fa5d011c180 con 0x7fa5d00704e0 2024-03-25T18:26:25.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.263+0000 7fa5c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.263+0000 7fa5c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.263+0000 7fa5c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.263+0000 7fa5c6ffd640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:26:25.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.263+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b4004eb0 con 0x7fa5d00704e0 2024-03-25T18:26:25.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.271+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477374174 0 0) 0x7fa5d013eab0 con 0x7fa5d011bcc0 2024-03-25T18:26:25.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.271+0000 7fa5c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.271+0000 7fa5c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.271+0000 7fa5c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.271+0000 7fa5c6ffd640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:26:25.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.271+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d011c180 con 0x7fa5d011bcc0 2024-03-25T18:26:25.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.272+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869712858 0 0) 0x7fa5b4004eb0 con 0x7fa5d00704e0 2024-03-25T18:26:25.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.272+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5d013eab0 con 0x7fa5d00704e0 2024-03-25T18:26:25.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.272+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa5cc001990 con 0x7fa5d00704e0 2024-03-25T18:26:25.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.272+0000 7fa5c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.272+0000 7fa5c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:25.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.272+0000 7fa5c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:25.274 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:25.275 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.272+0000 7fa5c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:25.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.274+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992344261 0 0) 0x7fa5d011d2f0 con 0x7fa5d00ae6e0 2024-03-25T18:26:25.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.274+0000 7fa5c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.274+0000 7fa5c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.274+0000 7fa5c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.274+0000 7fa5c6ffd640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:26:25.276 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.274+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b4004eb0 con 0x7fa5d00ae6e0 2024-03-25T18:26:25.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.282+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3288387903 0 0) 0x7fa5d013eab0 con 0x7fa5d00704e0 2024-03-25T18:26:25.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.282+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 >> v1:172.21.15.42:6790/0 conn(0x7fa5d011bcc0 legacy=0x7fa5d01d7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 >> v1:172.21.15.42:6789/0 conn(0x7fa5d00ae6e0 legacy=0x7fa5d01d73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:25.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 1 monclient: found mon.b 2024-03-25T18:26:25.284 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d013ec40 con 0x7fa5d00704e0 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 10 monclient: _renew_subs 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d0142c30 con 0x7fa5d00704e0 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 10 monclient: _renew_subs 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:25.285 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d01377a0 con 0x7fa5d00704e0 2024-03-25T18:26:25.286 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.283+0000 7fa5d8739640 1 librados: init done 2024-03-25T18:26:25.286 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.284+0000 7fa5d8739640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.286 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.284+0000 7fa5d8739640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.286 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.284+0000 7fa5d8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:25.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.284+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d0142fd0 con 0x7fa5d00704e0 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.551+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5cc003610 con 0x7fa5d00704e0 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.551+0000 7fa5c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.551+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa5cc0039e0 con 0x7fa5d00704e0 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.551+0000 7fa5c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.551+0000 7fa5c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.551+0000 7fa5c6ffd640 10 monclient: dump: 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:25.553 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.551+0000 7fa5d4cd2640 4 set_mon_vals no callback set 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.552+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fa5cc003c00 con 0x7fa5d00704e0 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.553+0000 7fa5d5cd4640 10 monclient: get_auth_request con 0x7fa5b406de20 auth_method 0 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.553+0000 7fa5c6ffd640 10 client.4537.objecter ms_handle_connect 0x7fa5b406de20 2024-03-25T18:26:25.558 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fa5cc054800 con 0x7fa5d00704e0 2024-03-25T18:26:25.558 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 10 client.4537.objecter ms_dispatch 0x7fa5d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:25.558 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:25.558 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:25.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-25T18:26:25.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5cc050310 con 0x7fa5d00704e0 2024-03-25T18:26:25.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.557+0000 7fa5c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:25.800 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.799+0000 7fa5d8739640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:25.800 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.799+0000 7fa5d8739640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:25.800 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.799+0000 7fa5d8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:25.800 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.799+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5d011c310 con 0x7fa5d00704e0 2024-03-25T18:26:25.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.819+0000 7fa5c6ffd640 1 -- 172.21.15.42:0/3917646342 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fa5cc052240 con 0x7fa5d00704e0 2024-03-25T18:26:25.821 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:25.821 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:25.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.819+0000 7fa5c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.819+0000 7fa5c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:25.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 10 librados: watch_flush enter 2024-03-25T18:26:25.823 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 10 librados: watch_flush exit 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 10 client.4537.objecter successfully canceled tick 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 >> v1:172.21.15.42:6800/3081750888 conn(0x7fa5b406de20 legacy=0x7fa5b40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 10 monclient: shutdown 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 >> v1:172.21.15.120:6789/0 conn(0x7fa5d00704e0 legacy=0x7fa5d0136eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 shutdown_connections 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 >> 172.21.15.42:0/3917646342 conn(0x7fa5d006b630 msgr2=0x7fa5d013c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 shutdown_connections 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 1 -- 172.21.15.42:0/3917646342 wait complete. 2024-03-25T18:26:25.824 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:25.822+0000 7fa5d8739640 1 librados: shutdown 2024-03-25T18:26:25.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:27.568 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:10.737 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.616 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:10.785 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.637 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:10.806 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.755 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:10.875 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.762 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:10.882 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.764 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:10.933 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.772 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:10.942 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.795 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:10.915 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:27.809 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:10.978 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.643 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:32.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.757+0000 7f32ba2fb640 10 monclient: get_monmap_and_config 2024-03-25T18:26:32.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.757+0000 7f32ba2fb640 10 monclient: build_initial_monmap 2024-03-25T18:26:32.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.757+0000 7f32ba2fb640 10 monclient: monmap: 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:32.758179+0000 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:32.758179+0000 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:32.759 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:32.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 1 Processor -- start 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 1 -- start start 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient: init 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient(hunting): picked mon.noname-c con 0x7f32b40a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient(hunting): picked mon.noname-b con 0x7f32b413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient(hunting): picked mon.noname-a con 0x7f32b41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b4142520 con 0x7f32b41430c0 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40b28a0 con 0x7f32b40a98f0 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40b2a30 con 0x7f32b413f710 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:32.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.758+0000 7f32ba2fb640 10 monclient(hunting): authenticate will time out at 7707.985346s 2024-03-25T18:26:32.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.770+0000 7f32b37fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f32b413f710 0x7f32b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:56230/0 (socket says 172.21.15.42:56230) 2024-03-25T18:26:32.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.770+0000 7f32b37fe640 1 -- 172.21.15.42:0/1179687202 learned_addr learned my addr 172.21.15.42:0/1179687202 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:32.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.784+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84529712 0 0) 0x7f32b40b2a30 con 0x7f32b413f710 2024-03-25T18:26:32.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.784+0000 7f32b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.784+0000 7f32b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.784+0000 7f32b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.784+0000 7f32b2ffd640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:26:32.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.784+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f329c003520 con 0x7f32b413f710 2024-03-25T18:26:32.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.787+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100934716 0 0) 0x7f32b4142520 con 0x7f32b41430c0 2024-03-25T18:26:32.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.787+0000 7f32b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.787+0000 7f32b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.787+0000 7f32b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.787+0000 7f32b2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:26:32.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.787+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b40b2a30 con 0x7f32b41430c0 2024-03-25T18:26:32.795 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 240337339 0 0) 0x7f329c003520 con 0x7f32b413f710 2024-03-25T18:26:32.795 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 >> v1:172.21.15.120:6789/0 conn(0x7f32b40a98f0 legacy=0x7f32b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 >> v1:172.21.15.42:6789/0 conn(0x7f32b41430c0 legacy=0x7f32b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b40b44c0 con 0x7f32b413f710 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f32a8001690 con 0x7f32b413f710 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.796 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32b2ffd640 10 monclient: dump: 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32ba2fb640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:26:32.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.793+0000 7f32ba2fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:32.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a80017f0 con 0x7f32b413f710 2024-03-25T18:26:32.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:32.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32b2ffd640 1 -- 172.21.15.42:0/1179687202 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f32a8001c00 con 0x7f32b413f710 2024-03-25T18:26:32.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32b2ffd640 10 monclient: dump: 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32ba2fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:32.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32ba2fb640 4 set_mon_vals no callback set 2024-03-25T18:26:32.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32ba2fb640 10 monclient: shutdown 2024-03-25T18:26:32.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32ba2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:32.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.805+0000 7f32ba2fb640 1 -- 172.21.15.42:0/1179687202 >> v1:172.21.15.42:6790/0 conn(0x7f32b413f710 legacy=0x7f32b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.806+0000 7f32ba2fb640 1 -- 172.21.15.42:0/1179687202 shutdown_connections 2024-03-25T18:26:32.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.806+0000 7f32ba2fb640 1 -- 172.21.15.42:0/1179687202 >> 172.21.15.42:0/1179687202 conn(0x7f32b40b7ea0 msgr2=0x7f32b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.806+0000 7f32ba2fb640 1 -- 172.21.15.42:0/1179687202 shutdown_connections 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.806+0000 7f32ba2fb640 1 -- 172.21.15.42:0/1179687202 wait complete. 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.806+0000 7f32ba2fb640 10 monclient: build_initial_monmap 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.806+0000 7f32ba2fb640 10 monclient: monmap: 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:32.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.807+0000 7f32ba2fb640 1 librados: starting msgr at 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.807+0000 7f32ba2fb640 1 librados: starting objecter 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.807+0000 7f32ba2fb640 1 Processor -- start 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.807+0000 7f32ba2fb640 1 -- start start 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.807+0000 7f32ba2fb640 1 librados: setting wanted keys 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.807+0000 7f32ba2fb640 1 librados: calling monclient init 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.807+0000 7f32ba2fb640 10 monclient: init 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 10 monclient(hunting): picked mon.noname-a con 0x7f32b40a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:32.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 10 monclient(hunting): picked mon.noname-c con 0x7f32b413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:32.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 10 monclient(hunting): picked mon.noname-b con 0x7f32b41430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:32.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40abc80 con 0x7f32b40a98f0 2024-03-25T18:26:32.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b41d7780 con 0x7f32b41430c0 2024-03-25T18:26:32.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b41399a0 con 0x7f32b413f710 2024-03-25T18:26:32.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:32.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.808+0000 7f32ba2fb640 10 monclient(hunting): authenticate will time out at 7708.035212s 2024-03-25T18:26:32.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.820+0000 7f32b37fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f32b413f710 0x7f32b41d4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:56288/0 (socket says 172.21.15.42:56288) 2024-03-25T18:26:32.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.820+0000 7f32b37fe640 1 -- 172.21.15.42:0/4221011669 learned_addr learned my addr 172.21.15.42:0/4221011669 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:32.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.826+0000 7f32b0ff9640 10 client.?.objecter ms_handle_connect 0x7f32b413f710 2024-03-25T18:26:32.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.826+0000 7f32b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:32.838 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.836+0000 7f32b0ff9640 10 client.?.objecter ms_handle_connect 0x7f32b41430c0 2024-03-25T18:26:32.838 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.836+0000 7f32b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:32.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.838+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558268720 0 0) 0x7f32b41399a0 con 0x7f32b413f710 2024-03-25T18:26:32.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.838+0000 7f32b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.838+0000 7f32b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.838+0000 7f32b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.838+0000 7f32b0ff9640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:26:32.839 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.838+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a0004eb0 con 0x7f32b413f710 2024-03-25T18:26:32.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 10 client.?.objecter ms_handle_connect 0x7f32b40a98f0 2024-03-25T18:26:32.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:32.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676874357 0 0) 0x7f32b41d7780 con 0x7f32b41430c0 2024-03-25T18:26:32.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.846 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:26:32.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.845+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b41399a0 con 0x7f32b41430c0 2024-03-25T18:26:32.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.863+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 888736742 0 0) 0x7f32a0004eb0 con 0x7f32b413f710 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.863+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b41d7780 con 0x7f32b413f710 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.863+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f32a8001c00 con 0x7f32b413f710 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.863+0000 7f32b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.863+0000 7f32b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.863+0000 7f32b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:32.865 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:32.866 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:32.866 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:32.866 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:32.866 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.863+0000 7f32b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.864+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599637158 0 0) 0x7f32b40abc80 con 0x7f32b40a98f0 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.864+0000 7f32b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.864+0000 7f32b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.864+0000 7f32b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.864+0000 7f32b0ff9640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.864+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a0004eb0 con 0x7f32b40a98f0 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.865+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2863787270 0 0) 0x7f32b41399a0 con 0x7f32b41430c0 2024-03-25T18:26:32.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.865+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b40abc80 con 0x7f32b41430c0 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.865+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f32ac001b90 con 0x7f32b41430c0 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.865+0000 7f32b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.865+0000 7f32b0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.865+0000 7f32b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:32.868 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:32.869 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:32.875 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.874+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1666141561 0 0) 0x7f32a0004eb0 con 0x7f32b40a98f0 2024-03-25T18:26:32.875 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.874+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b41399a0 con 0x7f32b40a98f0 2024-03-25T18:26:32.875 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.874+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f32a4001860 con 0x7f32b40a98f0 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.874+0000 7f32b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.874+0000 7f32b0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.874+0000 7f32b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:32.876 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:32.877 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:32.877 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:32.877 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:32.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3834384053 0 0) 0x7f32b41d7780 con 0x7f32b413f710 2024-03-25T18:26:32.879 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 >> v1:172.21.15.120:6789/0 conn(0x7f32b41430c0 legacy=0x7f32b4138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 >> v1:172.21.15.42:6789/0 conn(0x7f32b40a98f0 legacy=0x7f32b41d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 1 monclient: found mon.c 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b41d41e0 con 0x7f32b413f710 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:32.880 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 10 monclient: _renew_subs 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b41d8370 con 0x7f32b413f710 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 10 monclient: _renew_subs 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b41d5130 con 0x7f32b413f710 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.877+0000 7f32ba2fb640 1 librados: init done 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.878+0000 7f32ba2fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.878+0000 7f32ba2fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.878+0000 7f32ba2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:26:32.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:32.878+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b40627b0 con 0x7f32b413f710 2024-03-25T18:26:33.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.191+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a8004a90 con 0x7f32b413f710 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.191+0000 7f32b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.191+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f32a8004e20 con 0x7f32b413f710 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.191+0000 7f32b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.191+0000 7f32b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.191+0000 7f32b0ff9640 10 monclient: dump: 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.193 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:33.194 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:33.194 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:33.194 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:33.194 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:33.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.191+0000 7f32b27fc640 4 set_mon_vals no callback set 2024-03-25T18:26:33.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.192+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f32a8005040 con 0x7f32b413f710 2024-03-25T18:26:33.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.192+0000 7f32b3fff640 10 monclient: get_auth_request con 0x7f32a006e160 auth_method 0 2024-03-25T18:26:33.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.193+0000 7f32b0ff9640 10 client.4538.objecter ms_handle_connect 0x7f32a006e160 2024-03-25T18:26:33.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f32a8054970 con 0x7f32b413f710 2024-03-25T18:26:33.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 10 client.4538.objecter ms_dispatch 0x7f32b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:33.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:33.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:33.202 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:26:33.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32a8050480 con 0x7f32b413f710 2024-03-25T18:26:33.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:33.203 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.201+0000 7f32b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:33.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.449+0000 7f32ba2fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.449+0000 7f32ba2fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.449+0000 7f32ba2fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:26:33.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.449+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32b41d7910 con 0x7f32b413f710 2024-03-25T18:26:33.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.475+0000 7f32b0ff9640 1 -- 172.21.15.42:0/4221011669 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f32a80523b0 con 0x7f32b413f710 2024-03-25T18:26:33.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.475+0000 7f32b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.475+0000 7f32b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:33.477 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:33.477 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:33.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.477+0000 7f32ba2fb640 10 librados: watch_flush enter 2024-03-25T18:26:33.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.477+0000 7f32ba2fb640 10 librados: watch_flush exit 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.477+0000 7f32ba2fb640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.477+0000 7f32ba2fb640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 >> v1:172.21.15.42:6800/3081750888 conn(0x7f32a006e160 legacy=0x7f32a0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 10 monclient: shutdown 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 >> v1:172.21.15.42:6790/0 conn(0x7f32b413f710 legacy=0x7f32b41d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 shutdown_connections 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 >> 172.21.15.42:0/4221011669 conn(0x7f32b40b7ea0 msgr2=0x7f32b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 shutdown_connections 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 1 -- 172.21.15.42:0/4221011669 wait complete. 2024-03-25T18:26:33.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:33.478+0000 7f32ba2fb640 1 librados: shutdown 2024-03-25T18:26:33.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:40.301 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.402+0000 7fdad142a640 10 monclient: get_monmap_and_config 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.402+0000 7fdad142a640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.402+0000 7fdad142a640 10 monclient: monmap: 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:40.403783+0000 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:40.403783+0000 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:40.404 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:40.405 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:40.405 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:40.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.403+0000 7fdad142a640 1 Processor -- start 2024-03-25T18:26:40.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.403+0000 7fdad142a640 1 -- start start 2024-03-25T18:26:40.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.403+0000 7fdad142a640 10 monclient: init 2024-03-25T18:26:40.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:40.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc0b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc14be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0af940 con 0x7fdacc14be90 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0b0a70 con 0x7fdacc0b1730 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0b0d50 con 0x7fdacc0b4d60 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdad142a640 10 monclient(hunting): authenticate will time out at 7715.630792s 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdac3fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fdacc0b1730 0x7fdacc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:60272/0 (socket says 172.21.15.42:60272) 2024-03-25T18:26:40.407 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.404+0000 7fdac3fff640 1 -- 172.21.15.42:0/2976552126 learned_addr learned my addr 172.21.15.42:0/2976552126 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:40.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.463+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315581389 0 0) 0x7fdacc0af940 con 0x7fdacc14be90 2024-03-25T18:26:40.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.463+0000 7fdaca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.463+0000 7fdaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.463+0000 7fdaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.463+0000 7fdaca7fc640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:26:40.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.463+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab8003520 con 0x7fdacc14be90 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885422802 0 0) 0x7fdab8003520 con 0x7fdacc14be90 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 >> v1:172.21.15.42:6790/0 conn(0x7fdacc0b4d60 legacy=0x7fdacc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 >> v1:172.21.15.120:6789/0 conn(0x7fdacc0b1730 legacy=0x7fdacc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 1 monclient: found mon.noname-a 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc0b1030 con 0x7fdacc14be90 2024-03-25T18:26:40.479 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fdac4001a10 con 0x7fdacc14be90 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdad142a640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdad142a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdaca7fc640 10 monclient: dump: 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.480 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:40.481 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:40.481 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:40.481 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:40.481 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:40.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.477+0000 7fdad142a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:40.488 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdac4001b70 con 0x7fdacc14be90 2024-03-25T18:26:40.488 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdaca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdaca7fc640 1 -- 172.21.15.42:0/2976552126 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fdac4001f80 con 0x7fdacc14be90 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdaca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdaca7fc640 10 monclient: dump: 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:40.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdad142a640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:40.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.487+0000 7fdad142a640 4 set_mon_vals no callback set 2024-03-25T18:26:40.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.488+0000 7fdad142a640 10 monclient: shutdown 2024-03-25T18:26:40.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.488+0000 7fdad142a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.488+0000 7fdad142a640 1 -- 172.21.15.42:0/2976552126 >> v1:172.21.15.42:6789/0 conn(0x7fdacc14be90 legacy=0x7fdacc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.488+0000 7fdad142a640 1 -- 172.21.15.42:0/2976552126 shutdown_connections 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.488+0000 7fdad142a640 1 -- 172.21.15.42:0/2976552126 >> 172.21.15.42:0/2976552126 conn(0x7fdacc0617a0 msgr2=0x7fdacc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.488+0000 7fdad142a640 1 -- 172.21.15.42:0/2976552126 shutdown_connections 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.488+0000 7fdad142a640 1 -- 172.21.15.42:0/2976552126 wait complete. 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 10 monclient: monmap: 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:40.492 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 1 librados: starting msgr at 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 1 librados: starting objecter 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 1 Processor -- start 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 1 -- start start 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 1 librados: setting wanted keys 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 1 librados: calling monclient init 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.489+0000 7fdad142a640 10 monclient: init 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc0b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:40.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc14be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:40.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc143900 con 0x7fdacc0b4d60 2024-03-25T18:26:40.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc143be0 con 0x7fdacc0b1730 2024-03-25T18:26:40.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc138720 con 0x7fdacc14be90 2024-03-25T18:26:40.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.490+0000 7fdad142a640 10 monclient(hunting): authenticate will time out at 7715.716944s 2024-03-25T18:26:40.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.497+0000 7fdacaffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fdacc0b1730 0x7fdacc138010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:60312/0 (socket says 172.21.15.42:60312) 2024-03-25T18:26:40.498 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.497+0000 7fdacaffd640 1 -- 172.21.15.42:0/4144053267 learned_addr learned my addr 172.21.15.42:0/4144053267 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:40.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.498+0000 7fdac37fe640 10 client.?.objecter ms_handle_connect 0x7fdacc0b1730 2024-03-25T18:26:40.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.498+0000 7fdac37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.511+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638070354 0 0) 0x7fdacc143be0 con 0x7fdacc0b1730 2024-03-25T18:26:40.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.511+0000 7fdac37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.511+0000 7fdac37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.511+0000 7fdac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.511+0000 7fdac37fe640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:26:40.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.511+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaac004eb0 con 0x7fdacc0b1730 2024-03-25T18:26:40.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.515+0000 7fdac37fe640 10 client.?.objecter ms_handle_connect 0x7fdacc14be90 2024-03-25T18:26:40.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.515+0000 7fdac37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.522+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 699334040 0 0) 0x7fdaac004eb0 con 0x7fdacc0b1730 2024-03-25T18:26:40.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.522+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdacc143be0 con 0x7fdacc0b1730 2024-03-25T18:26:40.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.522+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fdabc0019d0 con 0x7fdacc0b1730 2024-03-25T18:26:40.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.522+0000 7fdac37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.522+0000 7fdac37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.522+0000 7fdac37fe640 10 monclient(hunting): dump: 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:40.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.522+0000 7fdac37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:40.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760333988 0 0) 0x7fdacc138720 con 0x7fdacc14be90 2024-03-25T18:26:40.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:26:40.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaac004eb0 con 0x7fdacc14be90 2024-03-25T18:26:40.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 10 client.?.objecter ms_handle_connect 0x7fdacc0b4d60 2024-03-25T18:26:40.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.525+0000 7fdac37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3702027375 0 0) 0x7fdacc143be0 con 0x7fdacc0b1730 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 >> v1:172.21.15.42:6790/0 conn(0x7fdacc14be90 legacy=0x7fdacc13a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 >> v1:172.21.15.42:6789/0 conn(0x7fdacc0b4d60 legacy=0x7fdacc139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 1 monclient: found mon.b 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc1388b0 con 0x7fdacc0b1730 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdac37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 5 monclient: authenticate success, global_id 4549 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 10 monclient: _renew_subs 2024-03-25T18:26:40.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc144070 con 0x7fdacc0b1730 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 10 monclient: _renew_subs 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc1445f0 con 0x7fdacc0b1730 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.527+0000 7fdad142a640 1 librados: init done 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.528+0000 7fdad142a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.528+0000 7fdad142a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.528+0000 7fdad142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:40.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.528+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc13bbd0 con 0x7fdacc0b1730 2024-03-25T18:26:40.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.807+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdabc001c90 con 0x7fdacc0b1730 2024-03-25T18:26:40.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.807+0000 7fdac37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:40.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.807+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fdabc003a60 con 0x7fdacc0b1730 2024-03-25T18:26:40.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.807+0000 7fdac37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.807+0000 7fdac37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:40.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.807+0000 7fdac37fe640 10 monclient: dump: 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:40.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.807+0000 7fdac9ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:40.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.808+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fdabc003c80 con 0x7fdacc0b1730 2024-03-25T18:26:40.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.809+0000 7fdac3fff640 10 monclient: get_auth_request con 0x7fdaac06de20 auth_method 0 2024-03-25T18:26:40.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.809+0000 7fdac37fe640 10 client.4549.objecter ms_handle_connect 0x7fdaac06de20 2024-03-25T18:26:40.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.814+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fdabc054840 con 0x7fdacc0b1730 2024-03-25T18:26:40.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.814+0000 7fdac37fe640 10 client.4549.objecter ms_dispatch 0x7fdacc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:40.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.814+0000 7fdac37fe640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:40.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.814+0000 7fdac37fe640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:40.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.814+0000 7fdac37fe640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T18:26:40.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.815+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdabc050350 con 0x7fdacc0b1730 2024-03-25T18:26:40.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.815+0000 7fdac37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.816 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:40.815+0000 7fdac37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:41.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.067+0000 7fdad142a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.067+0000 7fdad142a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.067+0000 7fdad142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:41.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.067+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdacc144200 con 0x7fdacc0b1730 2024-03-25T18:26:41.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.086+0000 7fdac37fe640 1 -- 172.21.15.42:0/4144053267 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fdabc052280 con 0x7fdacc0b1730 2024-03-25T18:26:41.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.086+0000 7fdac37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.087 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.086+0000 7fdac37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:41.088 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:41.088 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 10 librados: watch_flush enter 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 10 librados: watch_flush exit 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 10 client.4549.objecter successfully canceled tick 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 >> v1:172.21.15.42:6800/3081750888 conn(0x7fdaac06de20 legacy=0x7fdaac0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 10 monclient: shutdown 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 >> v1:172.21.15.120:6789/0 conn(0x7fdacc0b1730 legacy=0x7fdacc138010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 shutdown_connections 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 >> 172.21.15.42:0/4144053267 conn(0x7fdacc0617a0 msgr2=0x7fdacc0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 shutdown_connections 2024-03-25T18:26:41.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 1 -- 172.21.15.42:0/4144053267 wait complete. 2024-03-25T18:26:41.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:41.090+0000 7fdad142a640 1 librados: shutdown 2024-03-25T18:26:41.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:47.676 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:30.845 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.683 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:30.852 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.714 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:30.883 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.737 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:30.858 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.741 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:30.910 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.743 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:30.864 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.746 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:30.916 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.786 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:30.955 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.796 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:30.917 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:47.911 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.020+0000 7f973b063640 10 monclient: get_monmap_and_config 2024-03-25T18:26:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.020+0000 7f973b063640 10 monclient: build_initial_monmap 2024-03-25T18:26:48.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.020+0000 7f973b063640 10 monclient: monmap: 2024-03-25T18:26:48.022 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:48.022 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:48.022 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:48.021825+0000 2024-03-25T18:26:48.022 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:48.021825+0000 2024-03-25T18:26:48.023 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:48.023 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:48.023 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:48.023 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:48.023 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:48.023 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:48.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.021+0000 7f973b063640 1 Processor -- start 2024-03-25T18:26:48.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.021+0000 7f973b063640 1 -- start start 2024-03-25T18:26:48.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.021+0000 7f973b063640 10 monclient: init 2024-03-25T18:26:48.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:48.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:48.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 10 monclient(hunting): picked mon.noname-a con 0x7f97340a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:48.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 10 monclient(hunting): picked mon.noname-b con 0x7f973413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:48.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 10 monclient(hunting): picked mon.noname-c con 0x7f97341430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:48.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9734142520 con 0x7f97340a98f0 2024-03-25T18:26:48.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340b28a0 con 0x7f97341430c0 2024-03-25T18:26:48.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340b2a30 con 0x7f973413f710 2024-03-25T18:26:48.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:48.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.022+0000 7f973b063640 10 monclient(hunting): authenticate will time out at 7723.249145s 2024-03-25T18:26:48.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.033+0000 7f9739600640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f97341430c0 0x7f973414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:56644/0 (socket says 172.21.15.42:56644) 2024-03-25T18:26:48.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.033+0000 7f9739600640 1 -- 172.21.15.42:0/1710594120 learned_addr learned my addr 172.21.15.42:0/1710594120 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:48.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.042+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708629954 0 0) 0x7f9734142520 con 0x7f97340a98f0 2024-03-25T18:26:48.043 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.042+0000 7f97337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.042+0000 7f97337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.042+0000 7f97337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.042+0000 7f97337fe640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:26:48.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.042+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9720003520 con 0x7f97340a98f0 2024-03-25T18:26:48.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.053+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513862937 0 0) 0x7f97340b28a0 con 0x7f97341430c0 2024-03-25T18:26:48.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.053+0000 7f97337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.053+0000 7f97337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.053+0000 7f97337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.055 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.053+0000 7f97337fe640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:26:48.055 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.053+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9734142520 con 0x7f97341430c0 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418582264 0 0) 0x7f9720003520 con 0x7f97340a98f0 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 >> v1:172.21.15.42:6790/0 conn(0x7f973413f710 legacy=0x7f97341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 >> v1:172.21.15.120:6789/0 conn(0x7f97341430c0 legacy=0x7f973414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 1 monclient: found mon.noname-a 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340b44c0 con 0x7f97340a98f0 2024-03-25T18:26:48.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9724001690 con 0x7f97340a98f0 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f97337fe640 10 monclient: dump: 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:48.059 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:48.060 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:48.060 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:48.060 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f973b063640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:26:48.060 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.056+0000 7f973b063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:48.069 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97240017f0 con 0x7f97340a98f0 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f97337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f97337fe640 1 -- 172.21.15.42:0/1710594120 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9724001c00 con 0x7f97340a98f0 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f97337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f97337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f97337fe640 10 monclient: dump: 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:48.070 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:48.071 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:48.071 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:48.071 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:48.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f973b063640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:48.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.068+0000 7f973b063640 4 set_mon_vals no callback set 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.069+0000 7f973b063640 10 monclient: shutdown 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.069+0000 7f973b063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.069+0000 7f973b063640 1 -- 172.21.15.42:0/1710594120 >> v1:172.21.15.42:6789/0 conn(0x7f97340a98f0 legacy=0x7f97340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.069+0000 7f973b063640 1 -- 172.21.15.42:0/1710594120 shutdown_connections 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.069+0000 7f973b063640 1 -- 172.21.15.42:0/1710594120 >> 172.21.15.42:0/1710594120 conn(0x7f97340b7ea0 msgr2=0x7f97340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.069+0000 7f973b063640 1 -- 172.21.15.42:0/1710594120 shutdown_connections 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.069+0000 7f973b063640 1 -- 172.21.15.42:0/1710594120 wait complete. 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 10 monclient: build_initial_monmap 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 10 monclient: monmap: 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:48.073 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 1 librados: starting msgr at 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 1 librados: starting objecter 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 1 Processor -- start 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 1 -- start start 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 1 librados: setting wanted keys 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 1 librados: calling monclient init 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.070+0000 7f973b063640 10 monclient: init 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:48.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 10 monclient(hunting): picked mon.noname-b con 0x7f97340a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 10 monclient(hunting): picked mon.noname-a con 0x7f973413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 10 monclient(hunting): picked mon.noname-c con 0x7f97341430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9734132a30 con 0x7f973413f710 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9734138c80 con 0x7f97340a98f0 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9734138f60 con 0x7f97341430c0 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:48.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.071+0000 7f973b063640 10 monclient(hunting): authenticate will time out at 7723.298071s 2024-03-25T18:26:48.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.082+0000 7f9738dff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f97340a98f0 0x7f9734137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:56688/0 (socket says 172.21.15.42:56688) 2024-03-25T18:26:48.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.082+0000 7f9738dff640 1 -- 172.21.15.42:0/4028401848 learned_addr learned my addr 172.21.15.42:0/4028401848 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:48.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.083+0000 7f97317fa640 10 client.?.objecter ms_handle_connect 0x7f97340a98f0 2024-03-25T18:26:48.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.083+0000 7f97317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.089+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891502255 0 0) 0x7f9734138c80 con 0x7f97340a98f0 2024-03-25T18:26:48.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.089+0000 7f97317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.089+0000 7f97317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.089+0000 7f97317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.089+0000 7f97317fa640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:26:48.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.089+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9714004eb0 con 0x7f97340a98f0 2024-03-25T18:26:48.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.093+0000 7f97317fa640 10 client.?.objecter ms_handle_connect 0x7f973413f710 2024-03-25T18:26:48.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.093+0000 7f97317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.097+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 48964158 0 0) 0x7f9714004eb0 con 0x7f97340a98f0 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.097+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9734138c80 con 0x7f97340a98f0 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.097+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9724001bc0 con 0x7f97340a98f0 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.097+0000 7f97317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.097+0000 7f97317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.097+0000 7f97317fa640 10 monclient(hunting): dump: 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:48.099 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:48.100 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:48.100 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:48.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.097+0000 7f97317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:48.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.098+0000 7f97317fa640 10 client.?.objecter ms_handle_connect 0x7f97341430c0 2024-03-25T18:26:48.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.098+0000 7f97317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.099+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844462304 0 0) 0x7f9734132a30 con 0x7f973413f710 2024-03-25T18:26:48.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.099+0000 7f97317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.099+0000 7f97317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.099+0000 7f97317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.099+0000 7f97317fa640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:26:48.102 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.099+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9714004eb0 con 0x7f973413f710 2024-03-25T18:26:48.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1496662599 0 0) 0x7f9734138c80 con 0x7f97340a98f0 2024-03-25T18:26:48.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 >> v1:172.21.15.42:6790/0 conn(0x7f97341430c0 legacy=0x7f9734132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 >> v1:172.21.15.42:6789/0 conn(0x7f973413f710 legacy=0x7f9734131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 1 monclient: found mon.b 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734139240 con 0x7f97340a98f0 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f97317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 10 monclient: _renew_subs 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:48.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9734136e50 con 0x7f97340a98f0 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 10 monclient: _renew_subs 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97341389f0 con 0x7f97340a98f0 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.102+0000 7f973b063640 1 librados: init done 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.103+0000 7f973b063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.103+0000 7f973b063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.103+0000 7f973b063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:48.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.103+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97340627b0 con 0x7f97340a98f0 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.381+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97240038a0 con 0x7f97340a98f0 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.381+0000 7f97317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.381+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f9724003cb0 con 0x7f97340a98f0 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.381+0000 7f97317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.381+0000 7f97317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.381+0000 7f97317fa640 10 monclient: dump: 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.383 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.381+0000 7f9732ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.382+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f9724003ed0 con 0x7f97340a98f0 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.383+0000 7f9733fff640 10 monclient: get_auth_request con 0x7f971406dd90 auth_method 0 2024-03-25T18:26:48.384 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.383+0000 7f97317fa640 10 client.4555.objecter ms_handle_connect 0x7f971406dd90 2024-03-25T18:26:48.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f97240548c0 con 0x7f97340a98f0 2024-03-25T18:26:48.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 10 client.4555.objecter ms_dispatch 0x7f9734000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:48.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:48.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:48.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T18:26:48.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97240503d0 con 0x7f97340a98f0 2024-03-25T18:26:48.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.387+0000 7f97317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:48.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.650+0000 7f973b063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.650+0000 7f973b063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.650+0000 7f973b063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:26:48.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.650+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f973413dd60 con 0x7f97340a98f0 2024-03-25T18:26:48.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.764+0000 7f97317fa640 1 -- 172.21.15.42:0/4028401848 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f9724052300 con 0x7f97340a98f0 2024-03-25T18:26:48.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.764+0000 7f97317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.764+0000 7f97317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:48.766 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:48.766 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:48.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 10 librados: watch_flush enter 2024-03-25T18:26:48.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 10 librados: watch_flush exit 2024-03-25T18:26:48.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T18:26:48.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 10 client.4555.objecter successfully canceled tick 2024-03-25T18:26:48.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 >> v1:172.21.15.42:6800/3081750888 conn(0x7f971406dd90 legacy=0x7f9714070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 10 monclient: shutdown 2024-03-25T18:26:48.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:48.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 >> v1:172.21.15.120:6789/0 conn(0x7f97340a98f0 legacy=0x7f9734137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 shutdown_connections 2024-03-25T18:26:48.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.766+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 >> 172.21.15.42:0/4028401848 conn(0x7f97340b7ea0 msgr2=0x7f9734074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:48.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.767+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 shutdown_connections 2024-03-25T18:26:48.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.767+0000 7f973b063640 1 -- 172.21.15.42:0/4028401848 wait complete. 2024-03-25T18:26:48.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:48.767+0000 7f973b063640 1 librados: shutdown 2024-03-25T18:26:48.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:55.588 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.697+0000 7fa6bc2d3640 10 monclient: get_monmap_and_config 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.697+0000 7fa6bc2d3640 10 monclient: build_initial_monmap 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.697+0000 7fa6bc2d3640 10 monclient: monmap: 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:26:55.698964+0000 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:26:55.698964+0000 2024-03-25T18:26:55.699 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:55.700 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:55.700 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:55.700 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:26:55.700 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:26:55.700 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:55.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.698+0000 7fa6bc2d3640 1 Processor -- start 2024-03-25T18:26:55.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.698+0000 7fa6bc2d3640 1 -- start start 2024-03-25T18:26:55.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.698+0000 7fa6bc2d3640 10 monclient: init 2024-03-25T18:26:55.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.699+0000 7fa6bc2d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.699+0000 7fa6bc2d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.699+0000 7fa6bc2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fa6b40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.699+0000 7fa6bc2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fa6b413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.699+0000 7fa6bc2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fa6b41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.699+0000 7fa6bc2d3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b4142520 con 0x7fa6b41430c0 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.699+0000 7fa6bc2d3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b40b28a0 con 0x7fa6b413f710 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.700+0000 7fa6bc2d3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b40b2a30 con 0x7fa6b40a98f0 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.700+0000 7fa6bc2d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:55.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.700+0000 7fa6bc2d3640 10 monclient(hunting): authenticate will time out at 7730.926637s 2024-03-25T18:26:55.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.710+0000 7fa6b986e640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa6b413f710 0x7fa6b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:33092/0 (socket says 172.21.15.42:33092) 2024-03-25T18:26:55.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.710+0000 7fa6b986e640 1 -- 172.21.15.42:0/645530713 learned_addr learned my addr 172.21.15.42:0/645530713 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:55.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.717+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221158500 0 0) 0x7fa6b4142520 con 0x7fa6b41430c0 2024-03-25T18:26:55.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.717+0000 7fa6b906d640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.717+0000 7fa6b906d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.717+0000 7fa6b906d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.717+0000 7fa6b906d640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:26:55.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.717+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0003520 con 0x7fa6b41430c0 2024-03-25T18:26:55.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.718+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725999560 0 0) 0x7fa6b40b28a0 con 0x7fa6b413f710 2024-03-25T18:26:55.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.718+0000 7fa6b906d640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.718+0000 7fa6b906d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.718+0000 7fa6b906d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.718+0000 7fa6b906d640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:26:55.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.718+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4142520 con 0x7fa6b413f710 2024-03-25T18:26:55.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2882527461 0 0) 0x7fa6a0003520 con 0x7fa6b41430c0 2024-03-25T18:26:55.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 >> v1:172.21.15.42:6790/0 conn(0x7fa6b40a98f0 legacy=0x7fa6b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 >> v1:172.21.15.120:6789/0 conn(0x7fa6b413f710 legacy=0x7fa6b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 10 monclient: _finish_hunting 0 2024-03-25T18:26:55.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 1 monclient: found mon.noname-a 2024-03-25T18:26:55.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:55.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b40b44c0 con 0x7fa6b41430c0 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 10 monclient: _finish_auth 0 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 10 monclient: _check_auth_tickets 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa6ac001a10 con 0x7fa6b41430c0 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6b906d640 10 monclient: dump: 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:55.733 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:55.734 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:55.734 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:55.734 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:55.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6bc2d3640 5 monclient: authenticate success, global_id 4551 2024-03-25T18:26:55.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.730+0000 7fa6bc2d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:55.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6ac001bf0 con 0x7fa6b41430c0 2024-03-25T18:26:55.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6b906d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:55.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6b906d640 1 -- 172.21.15.42:0/645530713 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa6ac002000 con 0x7fa6b41430c0 2024-03-25T18:26:55.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6bc2d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:55.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6bc2d3640 4 set_mon_vals no callback set 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6b906d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6b906d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6b906d640 10 monclient: dump: 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:55.744 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:55.745 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6bc2d3640 10 monclient: shutdown 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6bc2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.742+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/645530713 >> v1:172.21.15.42:6789/0 conn(0x7fa6b41430c0 legacy=0x7fa6b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.743+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/645530713 shutdown_connections 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.743+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/645530713 >> 172.21.15.42:0/645530713 conn(0x7fa6b40b7ea0 msgr2=0x7fa6b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.743+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/645530713 shutdown_connections 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.743+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/645530713 wait complete. 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 10 monclient: build_initial_monmap 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 10 monclient: monmap: 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:55.747 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 1 librados: starting msgr at 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 1 librados: starting objecter 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 1 Processor -- start 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 1 -- start start 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 1 librados: setting wanted keys 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 1 librados: calling monclient init 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.744+0000 7fa6bc2d3640 10 monclient: init 2024-03-25T18:26:55.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fa6b40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fa6b413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fa6b41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b41db210 con 0x7fa6b41430c0 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b41d48f0 con 0x7fa6b413f710 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b414dc50 con 0x7fa6b40a98f0 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:55.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.745+0000 7fa6bc2d3640 10 monclient(hunting): authenticate will time out at 7730.972171s 2024-03-25T18:26:55.754 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.753+0000 7fa6ba06f640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fa6b40a98f0 0x7fa6b41dab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:44810/0 (socket says 172.21.15.42:44810) 2024-03-25T18:26:55.754 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.753+0000 7fa6ba06f640 1 -- 172.21.15.42:0/1824335887 learned_addr learned my addr 172.21.15.42:0/1824335887 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:26:55.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.761+0000 7fa6aaffd640 10 client.?.objecter ms_handle_connect 0x7fa6b41430c0 2024-03-25T18:26:55.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.761+0000 7fa6aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.763+0000 7fa6aaffd640 10 client.?.objecter ms_handle_connect 0x7fa6b40a98f0 2024-03-25T18:26:55.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.763+0000 7fa6aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.767+0000 7fa6aaffd640 10 client.?.objecter ms_handle_connect 0x7fa6b413f710 2024-03-25T18:26:55.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.767+0000 7fa6aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.774+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868140789 0 0) 0x7fa6b40b44c0 con 0x7fa6b41430c0 2024-03-25T18:26:55.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.774+0000 7fa6aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.776 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.774+0000 7fa6aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.776 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.774+0000 7fa6aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.776 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.774+0000 7fa6aaffd640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:26:55.776 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.774+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698004eb0 con 0x7fa6b41430c0 2024-03-25T18:26:55.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.780+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292384368 0 0) 0x7fa6b41d48f0 con 0x7fa6b413f710 2024-03-25T18:26:55.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.780+0000 7fa6aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.780+0000 7fa6aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.780+0000 7fa6aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.780+0000 7fa6aaffd640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:26:55.781 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.780+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b40b44c0 con 0x7fa6b413f710 2024-03-25T18:26:55.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.785+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528003617 0 0) 0x7fa6b414dc50 con 0x7fa6b40a98f0 2024-03-25T18:26:55.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.785+0000 7fa6aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.785+0000 7fa6aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.785+0000 7fa6aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.785+0000 7fa6aaffd640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:26:55.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.785+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b41d48f0 con 0x7fa6b40a98f0 2024-03-25T18:26:55.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.786+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3788964093 0 0) 0x7fa698004eb0 con 0x7fa6b41430c0 2024-03-25T18:26:55.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.786+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b414dc50 con 0x7fa6b41430c0 2024-03-25T18:26:55.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.786+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa6ac001d60 con 0x7fa6b41430c0 2024-03-25T18:26:55.787 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.786+0000 7fa6aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.786+0000 7fa6aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.786+0000 7fa6aaffd640 10 monclient(hunting): dump: 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:55.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.786+0000 7fa6aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3854537081 0 0) 0x7fa6b414dc50 con 0x7fa6b41430c0 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 >> v1:172.21.15.42:6790/0 conn(0x7fa6b40a98f0 legacy=0x7fa6b41dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 >> v1:172.21.15.120:6789/0 conn(0x7fa6b413f710 legacy=0x7fa6b41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 1 monclient: found mon.a 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b41db690 con 0x7fa6b41430c0 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:55.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 10 monclient: _renew_subs 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b41db960 con 0x7fa6b41430c0 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 10 monclient: _renew_subs 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b414e130 con 0x7fa6b41430c0 2024-03-25T18:26:55.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.804+0000 7fa6bc2d3640 1 librados: init done 2024-03-25T18:26:55.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.805+0000 7fa6bc2d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:55.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.805+0000 7fa6bc2d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:55.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.805+0000 7fa6bc2d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:26:55.808 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:55.805+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b41db3b0 con 0x7fa6b41430c0 2024-03-25T18:26:56.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.131+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6ac0015b0 con 0x7fa6b41430c0 2024-03-25T18:26:56.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.131+0000 7fa6aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:56.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.131+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa6ac003970 con 0x7fa6b41430c0 2024-03-25T18:26:56.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.131+0000 7fa6aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:56.133 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.131+0000 7fa6aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.131+0000 7fa6aaffd640 10 monclient: dump: 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:26:56.134 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.131+0000 7fa6b886c640 4 set_mon_vals no callback set 2024-03-25T18:26:56.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.132+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fa6ac019a00 con 0x7fa6b41430c0 2024-03-25T18:26:56.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.133+0000 7fa6ba06f640 10 monclient: get_auth_request con 0x7fa69806dda0 auth_method 0 2024-03-25T18:26:56.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.133+0000 7fa6aaffd640 10 client.4554.objecter ms_handle_connect 0x7fa69806dda0 2024-03-25T18:26:56.154 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.152+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fa6ac0536a0 con 0x7fa6b41430c0 2024-03-25T18:26:56.154 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.152+0000 7fa6aaffd640 10 client.4554.objecter ms_dispatch 0x7fa6b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:56.154 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.152+0000 7fa6aaffd640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:56.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.152+0000 7fa6aaffd640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:56.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.152+0000 7fa6aaffd640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-25T18:26:56.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.152+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6ac04efe0 con 0x7fa6b41430c0 2024-03-25T18:26:56.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.153+0000 7fa6aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:56.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.153+0000 7fa6aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:56.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.373+0000 7fa6bc2d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.373+0000 7fa6bc2d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.373+0000 7fa6bc2d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:26:56.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.373+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6b40627b0 con 0x7fa6b41430c0 2024-03-25T18:26:56.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.396+0000 7fa6aaffd640 1 -- 172.21.15.42:0/1824335887 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fa6ac0510e0 con 0x7fa6b41430c0 2024-03-25T18:26:56.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.396+0000 7fa6aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.396+0000 7fa6aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:56.398 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:26:56.398 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 10 librados: watch_flush enter 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 10 librados: watch_flush exit 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 10 client.4554.objecter successfully canceled tick 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 >> v1:172.21.15.42:6800/3081750888 conn(0x7fa69806dda0 legacy=0x7fa698070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 10 monclient: shutdown 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 >> v1:172.21.15.42:6789/0 conn(0x7fa6b41430c0 legacy=0x7fa6b414d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 shutdown_connections 2024-03-25T18:26:56.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 >> 172.21.15.42:0/1824335887 conn(0x7fa6b40b7ea0 msgr2=0x7fa6b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:56.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 shutdown_connections 2024-03-25T18:26:56.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 1 -- 172.21.15.42:0/1824335887 wait complete. 2024-03-25T18:26:56.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:26:56.398+0000 7fa6bc2d3640 1 librados: shutdown 2024-03-25T18:26:56.411 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:03.219 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:03.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.332+0000 7f76ef10d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.332+0000 7f76ef10d640 10 monclient: build_initial_monmap 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.332+0000 7f76ef10d640 10 monclient: monmap: 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:03.333568+0000 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:03.333568+0000 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:03.334 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:03.335 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:03.335 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:03.335 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:03.335 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:03.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.333+0000 7f76ef10d640 1 Processor -- start 2024-03-25T18:27:03.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.333+0000 7f76ef10d640 1 -- start start 2024-03-25T18:27:03.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.333+0000 7f76ef10d640 10 monclient: init 2024-03-25T18:27:03.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:03.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:03.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 10 monclient(hunting): picked mon.noname-b con 0x7f76e80b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:03.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 10 monclient(hunting): picked mon.noname-c con 0x7f76e80b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:03.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 10 monclient(hunting): picked mon.noname-a con 0x7f76e814be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:03.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80af940 con 0x7f76e814be90 2024-03-25T18:27:03.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80b0a70 con 0x7f76e80b1730 2024-03-25T18:27:03.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80b0d50 con 0x7f76e80b4d60 2024-03-25T18:27:03.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:03.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.334+0000 7f76ef10d640 10 monclient(hunting): authenticate will time out at 7738.560691s 2024-03-25T18:27:03.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.339+0000 7f76ed6aa640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f76e814be90 0x7f76e814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46226/0 (socket says 172.21.15.42:46226) 2024-03-25T18:27:03.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.340+0000 7f76ed6aa640 1 -- 172.21.15.42:0/1574592080 learned_addr learned my addr 172.21.15.42:0/1574592080 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:03.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.355+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148977935 0 0) 0x7f76e80af940 con 0x7f76e814be90 2024-03-25T18:27:03.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.355+0000 7f76e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.355+0000 7f76e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.355+0000 7f76e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.355+0000 7f76e77fe640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:27:03.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.355+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d8003520 con 0x7f76e814be90 2024-03-25T18:27:03.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.363+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149669047 0 0) 0x7f76e80b0d50 con 0x7f76e80b4d60 2024-03-25T18:27:03.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.363+0000 7f76e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.363+0000 7f76e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.363+0000 7f76e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.363+0000 7f76e77fe640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:27:03.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.363+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e80af940 con 0x7f76e80b4d60 2024-03-25T18:27:03.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.365+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628465246 0 0) 0x7f76e80b0a70 con 0x7f76e80b1730 2024-03-25T18:27:03.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.365+0000 7f76e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.365+0000 7f76e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.365+0000 7f76e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.365+0000 7f76e77fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:27:03.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.365+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e80b0d50 con 0x7f76e80b1730 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991871901 0 0) 0x7f76d8003520 con 0x7f76e814be90 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 >> v1:172.21.15.42:6790/0 conn(0x7f76e80b4d60 legacy=0x7f76e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 >> v1:172.21.15.120:6789/0 conn(0x7f76e80b1730 legacy=0x7f76e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 1 monclient: found mon.noname-a 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e80b1030 con 0x7f76e814be90 2024-03-25T18:27:03.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f76e0001a10 con 0x7f76e814be90 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76e77fe640 10 monclient: dump: 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:03.369 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:03.370 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:03.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76ef10d640 5 monclient: authenticate success, global_id 4557 2024-03-25T18:27:03.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.366+0000 7f76ef10d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e0001b70 con 0x7f76e814be90 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76e77fe640 1 -- 172.21.15.42:0/1574592080 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f76e0001f80 con 0x7f76e814be90 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76ef10d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76ef10d640 4 set_mon_vals no callback set 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.387+0000 7f76e77fe640 10 monclient: dump: 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:03.389 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:03.390 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.388+0000 7f76ef10d640 10 monclient: shutdown 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.388+0000 7f76ef10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.388+0000 7f76ef10d640 1 -- 172.21.15.42:0/1574592080 >> v1:172.21.15.42:6789/0 conn(0x7f76e814be90 legacy=0x7f76e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.388+0000 7f76ef10d640 1 -- 172.21.15.42:0/1574592080 shutdown_connections 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.388+0000 7f76ef10d640 1 -- 172.21.15.42:0/1574592080 >> 172.21.15.42:0/1574592080 conn(0x7f76e80617a0 msgr2=0x7f76e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.388+0000 7f76ef10d640 1 -- 172.21.15.42:0/1574592080 shutdown_connections 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.388+0000 7f76ef10d640 1 -- 172.21.15.42:0/1574592080 wait complete. 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 10 monclient: build_initial_monmap 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 10 monclient: monmap: 2024-03-25T18:27:03.392 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 1 librados: starting msgr at 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 1 librados: starting objecter 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 1 Processor -- start 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 1 -- start start 2024-03-25T18:27:03.393 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 1 librados: setting wanted keys 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 1 librados: calling monclient init 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.389+0000 7f76ef10d640 10 monclient: init 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 10 monclient(hunting): picked mon.noname-b con 0x7f76e80b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 10 monclient(hunting): picked mon.noname-a con 0x7f76e80b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 10 monclient(hunting): picked mon.noname-c con 0x7f76e814be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e8136610 con 0x7f76e80b4d60 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e8137db0 con 0x7f76e80b1730 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e8136210 con 0x7f76e814be90 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:03.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.390+0000 7f76ef10d640 10 monclient(hunting): authenticate will time out at 7738.617464s 2024-03-25T18:27:03.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.405+0000 7f76e7fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f76e80b4d60 0x7f76e813b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46262/0 (socket says 172.21.15.42:46262) 2024-03-25T18:27:03.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.405+0000 7f76e7fff640 1 -- 172.21.15.42:0/1682146896 learned_addr learned my addr 172.21.15.42:0/1682146896 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:03.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.413+0000 7f76e57fa640 10 client.?.objecter ms_handle_connect 0x7f76e80b4d60 2024-03-25T18:27:03.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.413+0000 7f76e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.413+0000 7f76e57fa640 10 client.?.objecter ms_handle_connect 0x7f76e814be90 2024-03-25T18:27:03.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.413+0000 7f76e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.417+0000 7f76e57fa640 10 client.?.objecter ms_handle_connect 0x7f76e80b1730 2024-03-25T18:27:03.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.417+0000 7f76e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.421+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537819183 0 0) 0x7f76e8136210 con 0x7f76e814be90 2024-03-25T18:27:03.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.421+0000 7f76e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.421+0000 7f76e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.421+0000 7f76e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.421+0000 7f76e57fa640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:27:03.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.421+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc004eb0 con 0x7f76e814be90 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.434+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389321740 0 0) 0x7f76e8136610 con 0x7f76e80b4d60 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.434+0000 7f76e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.434+0000 7f76e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.434+0000 7f76e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.434+0000 7f76e57fa640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.434+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e8136210 con 0x7f76e80b4d60 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.434+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2476542739 0 0) 0x7f76cc004eb0 con 0x7f76e814be90 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.435+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e8136610 con 0x7f76e814be90 2024-03-25T18:27:03.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.435+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f76e0001e80 con 0x7f76e814be90 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.435+0000 7f76e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.435+0000 7f76e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.435+0000 7f76e57fa640 10 monclient(hunting): dump: 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:03.437 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.435+0000 7f76e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:03.438 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.436+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899062428 0 0) 0x7f76e8137db0 con 0x7f76e80b1730 2024-03-25T18:27:03.438 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.436+0000 7f76e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.438 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.436+0000 7f76e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.438 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.436+0000 7f76e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.438 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.436+0000 7f76e57fa640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:27:03.438 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.436+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc004eb0 con 0x7f76e80b1730 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1144468936 0 0) 0x7f76e8136610 con 0x7f76e814be90 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 >> v1:172.21.15.120:6789/0 conn(0x7f76e80b1730 legacy=0x7f76e8135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 >> v1:172.21.15.42:6789/0 conn(0x7f76e80b4d60 legacy=0x7f76e813b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 1 monclient: found mon.c 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:03.457 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e813b9c0 con 0x7f76e814be90 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 10 monclient: _renew_subs 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e813bc10 con 0x7f76e814be90 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 10 monclient: _renew_subs 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e813ef00 con 0x7f76e814be90 2024-03-25T18:27:03.458 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.455+0000 7f76ef10d640 1 librados: init done 2024-03-25T18:27:03.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.456+0000 7f76ef10d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.456+0000 7f76ef10d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.456+0000 7f76ef10d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:03.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.456+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e80627b0 con 0x7f76e814be90 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.755+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e0001fe0 con 0x7f76e814be90 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.755+0000 7f76e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.755+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f76e0001420 con 0x7f76e814be90 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.755+0000 7f76e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.755+0000 7f76e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.755+0000 7f76e57fa640 10 monclient: dump: 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:03.757 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:03.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.755+0000 7f76e6ffd640 4 set_mon_vals no callback set 2024-03-25T18:27:03.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.756+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f76e001ae20 con 0x7f76e814be90 2024-03-25T18:27:03.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.757+0000 7f76ecea9640 10 monclient: get_auth_request con 0x7f76cc06dda0 auth_method 0 2024-03-25T18:27:03.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.757+0000 7f76e57fa640 10 client.4562.objecter ms_handle_connect 0x7f76cc06dda0 2024-03-25T18:27:03.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f76e00549e0 con 0x7f76e814be90 2024-03-25T18:27:03.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 10 client.4562.objecter ms_dispatch 0x7f76e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:03.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:03.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:03.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T18:27:03.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76e0050320 con 0x7f76e814be90 2024-03-25T18:27:03.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:03.764+0000 7f76e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:04.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.021+0000 7f76ef10d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.021+0000 7f76ef10d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.021+0000 7f76ef10d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:04.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.021+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76e81367c0 con 0x7f76e814be90 2024-03-25T18:27:04.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.046+0000 7f76e57fa640 1 -- 172.21.15.42:0/1682146896 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f76e81367c0 con 0x7f76e814be90 2024-03-25T18:27:04.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.046+0000 7f76e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.046+0000 7f76e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:04.048 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:04.048 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:04.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 10 librados: watch_flush enter 2024-03-25T18:27:04.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 10 librados: watch_flush exit 2024-03-25T18:27:04.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 10 client.4562.objecter successfully canceled tick 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 >> v1:172.21.15.42:6800/3081750888 conn(0x7f76cc06dda0 legacy=0x7f76cc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 10 monclient: shutdown 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 >> v1:172.21.15.42:6790/0 conn(0x7f76e814be90 legacy=0x7f76e81dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 shutdown_connections 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 >> 172.21.15.42:0/1682146896 conn(0x7f76e80617a0 msgr2=0x7f76e813c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 shutdown_connections 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 1 -- 172.21.15.42:0/1682146896 wait complete. 2024-03-25T18:27:04.051 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:04.049+0000 7f76ef10d640 1 librados: shutdown 2024-03-25T18:27:04.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:07.745 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:50.914 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.756 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:50.925 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.791 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:07:50.961 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.797 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:50.917 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.808 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:50.928 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.811 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:50.980 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.825 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:50.994 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.842 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:07:50.962 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:07.867 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:07:51.037 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:10.870 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.986+0000 7ff10a746640 10 monclient: get_monmap_and_config 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.986+0000 7ff10a746640 10 monclient: build_initial_monmap 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.986+0000 7ff10a746640 10 monclient: monmap: 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:10.987974+0000 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:10.987974+0000 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:10.988 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:10.989 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:10.989 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:10.989 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:10.989 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:10.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.987+0000 7ff10a746640 1 Processor -- start 2024-03-25T18:27:10.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.987+0000 7ff10a746640 1 -- start start 2024-03-25T18:27:10.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.987+0000 7ff10a746640 10 monclient: init 2024-03-25T18:27:10.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 10 monclient(hunting): picked mon.noname-a con 0x7ff1040b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 10 monclient(hunting): picked mon.noname-b con 0x7ff1040b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 10 monclient(hunting): picked mon.noname-c con 0x7ff10414be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1040af940 con 0x7ff1040b4d60 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1040b0a70 con 0x7ff10414be90 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1040b0d50 con 0x7ff1040b1730 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:10.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:10.988+0000 7ff10a746640 10 monclient(hunting): authenticate will time out at 7746.215466s 2024-03-25T18:27:11.002 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.001+0000 7ff1037fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ff1040b1730 0x7ff1040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:33124/0 (socket says 172.21.15.42:33124) 2024-03-25T18:27:11.002 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.001+0000 7ff1037fe640 1 -- 172.21.15.42:0/467584873 learned_addr learned my addr 172.21.15.42:0/467584873 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:11.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.013+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608128710 0 0) 0x7ff1040af940 con 0x7ff1040b4d60 2024-03-25T18:27:11.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.013+0000 7ff102ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.013+0000 7ff102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.013+0000 7ff102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.013+0000 7ff102ffd640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:27:11.015 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.013+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ec003520 con 0x7ff1040b4d60 2024-03-25T18:27:11.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.015+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106103257 0 0) 0x7ff1040b0a70 con 0x7ff10414be90 2024-03-25T18:27:11.016 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.015+0000 7ff102ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.015+0000 7ff102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.015+0000 7ff102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.015+0000 7ff102ffd640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:27:11.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.015+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1040af940 con 0x7ff10414be90 2024-03-25T18:27:11.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.016+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651665638 0 0) 0x7ff1040b0d50 con 0x7ff1040b1730 2024-03-25T18:27:11.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.016+0000 7ff102ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.016+0000 7ff102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.016+0000 7ff102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.016+0000 7ff102ffd640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:27:11.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.016+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1040b0a70 con 0x7ff1040b1730 2024-03-25T18:27:11.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2220060695 0 0) 0x7ff1040b0a70 con 0x7ff1040b1730 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 >> v1:172.21.15.120:6789/0 conn(0x7ff10414be90 legacy=0x7ff10414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 >> v1:172.21.15.42:6789/0 conn(0x7ff1040b4d60 legacy=0x7ff10413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 1 monclient: found mon.noname-b 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1040b1030 con 0x7ff1040b1730 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff0f80015b0 con 0x7ff1040b1730 2024-03-25T18:27:11.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff10a746640 5 monclient: authenticate success, global_id 4565 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff10a746640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff102ffd640 10 monclient: dump: 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:11.027 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.024+0000 7ff10a746640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0f8001710 con 0x7ff1040b1730 2024-03-25T18:27:11.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff102ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff102ffd640 1 -- 172.21.15.42:0/467584873 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff0f8001b60 con 0x7ff1040b1730 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff102ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff102ffd640 10 monclient: dump: 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:11.029 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:11.030 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:11.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff10a746640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:11.030 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff10a746640 4 set_mon_vals no callback set 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.027+0000 7ff10a746640 10 monclient: shutdown 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 1 -- 172.21.15.42:0/467584873 >> v1:172.21.15.42:6790/0 conn(0x7ff1040b1730 legacy=0x7ff1040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 1 -- 172.21.15.42:0/467584873 shutdown_connections 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 1 -- 172.21.15.42:0/467584873 >> 172.21.15.42:0/467584873 conn(0x7ff1040617a0 msgr2=0x7ff104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 1 -- 172.21.15.42:0/467584873 shutdown_connections 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 1 -- 172.21.15.42:0/467584873 wait complete. 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 10 monclient: build_initial_monmap 2024-03-25T18:27:11.032 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.028+0000 7ff10a746640 10 monclient: monmap: 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.029+0000 7ff10a746640 1 librados: starting msgr at 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.029+0000 7ff10a746640 1 librados: starting objecter 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.029+0000 7ff10a746640 1 Processor -- start 2024-03-25T18:27:11.033 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.029+0000 7ff10a746640 1 -- start start 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.029+0000 7ff10a746640 1 librados: setting wanted keys 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.029+0000 7ff10a746640 1 librados: calling monclient init 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.029+0000 7ff10a746640 10 monclient: init 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 10 monclient(hunting): picked mon.noname-a con 0x7ff1040b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 10 monclient(hunting): picked mon.noname-b con 0x7ff1040b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 10 monclient(hunting): picked mon.noname-c con 0x7ff10414be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1041409c0 con 0x7ff1040b1730 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff104141d10 con 0x7ff1040b4d60 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff104140e50 con 0x7ff10414be90 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:11.034 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.030+0000 7ff10a746640 10 monclient(hunting): authenticate will time out at 7746.256968s 2024-03-25T18:27:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.041+0000 7ff108ce3640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ff10414be90 0x7ff1041e2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:33150/0 (socket says 172.21.15.42:33150) 2024-03-25T18:27:11.042 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.041+0000 7ff108ce3640 1 -- 172.21.15.42:0/3900347409 learned_addr learned my addr 172.21.15.42:0/3900347409 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:11.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.045+0000 7ff100ff9640 10 client.?.objecter ms_handle_connect 0x7ff10414be90 2024-03-25T18:27:11.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.045+0000 7ff100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.053+0000 7ff100ff9640 10 client.?.objecter ms_handle_connect 0x7ff1040b1730 2024-03-25T18:27:11.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.053+0000 7ff100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.057+0000 7ff100ff9640 10 client.?.objecter ms_handle_connect 0x7ff1040b4d60 2024-03-25T18:27:11.058 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.057+0000 7ff100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.062+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311645273 0 0) 0x7ff104140e50 con 0x7ff10414be90 2024-03-25T18:27:11.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.062+0000 7ff100ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.062+0000 7ff100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.062+0000 7ff100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.062+0000 7ff100ff9640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:27:11.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.062+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f0004eb0 con 0x7ff10414be90 2024-03-25T18:27:11.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.069+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444795060 0 0) 0x7ff1041409c0 con 0x7ff1040b1730 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.069+0000 7ff100ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.069+0000 7ff100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.069+0000 7ff100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.069+0000 7ff100ff9640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.069+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff104140e50 con 0x7ff1040b1730 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.071+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207461292 0 0) 0x7ff104141d10 con 0x7ff1040b4d60 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.071+0000 7ff100ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.071+0000 7ff100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.071+0000 7ff100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.071+0000 7ff100ff9640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:27:11.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.071+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1041409c0 con 0x7ff1040b4d60 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.087+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1098410957 0 0) 0x7ff0f0004eb0 con 0x7ff10414be90 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.087+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff104141d10 con 0x7ff10414be90 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.087+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff0fc001c90 con 0x7ff10414be90 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.088+0000 7ff100ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.088+0000 7ff100ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.088+0000 7ff100ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:11.089 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:11.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.088+0000 7ff100ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:11.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.104+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4278836380 0 0) 0x7ff1041409c0 con 0x7ff1040b4d60 2024-03-25T18:27:11.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.104+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0f0004eb0 con 0x7ff1040b4d60 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.104+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff0f8001a20 con 0x7ff1040b4d60 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.104+0000 7ff100ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.104+0000 7ff100ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.104+0000 7ff100ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:11.106 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:11.107 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:11.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 414734532 0 0) 0x7ff104140e50 con 0x7ff1040b1730 2024-03-25T18:27:11.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1041409c0 con 0x7ff1040b1730 2024-03-25T18:27:11.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff0f4001950 con 0x7ff1040b1730 2024-03-25T18:27:11.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:11.109 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2884027666 0 0) 0x7ff104141d10 con 0x7ff10414be90 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 >> v1:172.21.15.120:6789/0 conn(0x7ff1040b4d60 legacy=0x7ff1041d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 >> v1:172.21.15.42:6789/0 conn(0x7ff1040b1730 legacy=0x7ff104131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 monclient: found mon.c 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff104132520 con 0x7ff10414be90 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:11.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff100ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 10 monclient: _renew_subs 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff104132770 con 0x7ff10414be90 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 10 monclient: _renew_subs 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff104138120 con 0x7ff10414be90 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.105+0000 7ff10a746640 1 librados: init done 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.106+0000 7ff10a746640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.106+0000 7ff10a746640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.106+0000 7ff10a746640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:11.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.106+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1040627b0 con 0x7ff10414be90 2024-03-25T18:27:11.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0fc003990 con 0x7ff10414be90 2024-03-25T18:27:11.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff100ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:11.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff0fc003d20 con 0x7ff10414be90 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff100ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff100ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff100ff9640 10 monclient: dump: 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:11.446 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:11.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff1027fc640 4 set_mon_vals no callback set 2024-03-25T18:27:11.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.444+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff0fc003f40 con 0x7ff10414be90 2024-03-25T18:27:11.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.445+0000 7ff103fff640 10 monclient: get_auth_request con 0x7ff0f006e0f0 auth_method 0 2024-03-25T18:27:11.447 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.445+0000 7ff100ff9640 10 client.4568.objecter ms_handle_connect 0x7ff0f006e0f0 2024-03-25T18:27:11.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7ff0fc0549e0 con 0x7ff10414be90 2024-03-25T18:27:11.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 10 client.4568.objecter ms_dispatch 0x7ff104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:11.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:11.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:11.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T18:27:11.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0fc0504f0 con 0x7ff10414be90 2024-03-25T18:27:11.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.452+0000 7ff100ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:11.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.685+0000 7ff10a746640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.685+0000 7ff10a746640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.685+0000 7ff10a746640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:11.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.685+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff104140b70 con 0x7ff10414be90 2024-03-25T18:27:11.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.720+0000 7ff100ff9640 1 -- 172.21.15.42:0/3900347409 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7ff0fc052420 con 0x7ff10414be90 2024-03-25T18:27:11.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.720+0000 7ff100ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.720+0000 7ff100ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:11.722 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:11.722 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:11.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.722+0000 7ff10a746640 10 librados: watch_flush enter 2024-03-25T18:27:11.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 10 librados: watch_flush exit 2024-03-25T18:27:11.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 10 client.4568.objecter successfully canceled tick 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff0f006e0f0 legacy=0x7ff0f00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 10 monclient: shutdown 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 >> v1:172.21.15.42:6790/0 conn(0x7ff10414be90 legacy=0x7ff1041e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 shutdown_connections 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 >> 172.21.15.42:0/3900347409 conn(0x7ff1040617a0 msgr2=0x7ff1040b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 shutdown_connections 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 1 -- 172.21.15.42:0/3900347409 wait complete. 2024-03-25T18:27:11.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:11.723+0000 7ff10a746640 1 librados: shutdown 2024-03-25T18:27:11.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:18.544 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:18.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.660+0000 7f7e49bf4640 10 monclient: get_monmap_and_config 2024-03-25T18:27:18.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.660+0000 7f7e49bf4640 10 monclient: build_initial_monmap 2024-03-25T18:27:18.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.660+0000 7f7e49bf4640 10 monclient: monmap: 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:18.661270+0000 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:18.661270+0000 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:18.662 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:18.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.660+0000 7f7e49bf4640 1 Processor -- start 2024-03-25T18:27:18.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 1 -- start start 2024-03-25T18:27:18.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient: init 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f7e440a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f7e4413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f7e441430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e44142520 con 0x7f7e4413f710 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440b28a0 con 0x7f7e441430c0 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440b2a30 con 0x7f7e440a98f0 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:18.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.661+0000 7f7e49bf4640 10 monclient(hunting): authenticate will time out at 7753.888335s 2024-03-25T18:27:18.673 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.671+0000 7f7e437fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f7e440a98f0 0x7f7e440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:54666/0 (socket says 172.21.15.42:54666) 2024-03-25T18:27:18.673 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.672+0000 7f7e437fe640 1 -- 172.21.15.42:0/3055904220 learned_addr learned my addr 172.21.15.42:0/3055904220 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:18.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.687+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581062170 0 0) 0x7f7e440b28a0 con 0x7f7e441430c0 2024-03-25T18:27:18.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.687+0000 7f7e427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.687+0000 7f7e427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.687+0000 7f7e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.687+0000 7f7e427fc640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:27:18.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.687+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c003520 con 0x7f7e441430c0 2024-03-25T18:27:18.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.691+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733633994 0 0) 0x7f7e440b2a30 con 0x7f7e440a98f0 2024-03-25T18:27:18.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.691+0000 7f7e427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.691+0000 7f7e427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.691+0000 7f7e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.691+0000 7f7e427fc640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:27:18.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.691+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e440b28a0 con 0x7f7e440a98f0 2024-03-25T18:27:18.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.695+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253235432 0 0) 0x7f7e44142520 con 0x7f7e4413f710 2024-03-25T18:27:18.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.695+0000 7f7e427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.695+0000 7f7e427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.695+0000 7f7e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.695+0000 7f7e427fc640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:27:18.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.695+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e440b2a30 con 0x7f7e4413f710 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1497647545 0 0) 0x7f7e2c003520 con 0x7f7e441430c0 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 >> v1:172.21.15.42:6790/0 conn(0x7f7e440a98f0 legacy=0x7f7e440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 >> v1:172.21.15.42:6789/0 conn(0x7f7e4413f710 legacy=0x7f7e441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 1 monclient: found mon.noname-c 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e440b44c0 con 0x7f7e441430c0 2024-03-25T18:27:18.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7e3c001a10 con 0x7f7e441430c0 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e49bf4640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e49bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.706+0000 7f7e427fc640 10 monclient: dump: 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:18.709 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:18.710 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:18.710 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:18.710 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:18.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.707+0000 7f7e49bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:18.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e3c001b70 con 0x7f7e441430c0 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e427fc640 1 -- 172.21.15.42:0/3055904220 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7e3c001f80 con 0x7f7e441430c0 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e49bf4640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e49bf4640 4 set_mon_vals no callback set 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.720+0000 7f7e427fc640 10 monclient: dump: 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:18.722 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.723 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:18.723 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:18.723 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:18.723 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:18.723 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.721+0000 7f7e49bf4640 10 monclient: shutdown 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.721+0000 7f7e49bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.721+0000 7f7e49bf4640 1 -- 172.21.15.42:0/3055904220 >> v1:172.21.15.120:6789/0 conn(0x7f7e441430c0 legacy=0x7f7e4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.721+0000 7f7e49bf4640 1 -- 172.21.15.42:0/3055904220 shutdown_connections 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.721+0000 7f7e49bf4640 1 -- 172.21.15.42:0/3055904220 >> 172.21.15.42:0/3055904220 conn(0x7f7e440b7ea0 msgr2=0x7f7e440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.721+0000 7f7e49bf4640 1 -- 172.21.15.42:0/3055904220 shutdown_connections 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.721+0000 7f7e49bf4640 1 -- 172.21.15.42:0/3055904220 wait complete. 2024-03-25T18:27:18.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.722+0000 7f7e49bf4640 10 monclient: build_initial_monmap 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.722+0000 7f7e49bf4640 10 monclient: monmap: 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.722+0000 7f7e49bf4640 1 librados: starting msgr at 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.722+0000 7f7e49bf4640 1 librados: starting objecter 2024-03-25T18:27:18.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.722+0000 7f7e49bf4640 1 Processor -- start 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 1 -- start start 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 1 librados: setting wanted keys 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 1 librados: calling monclient init 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient: init 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f7e440a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f7e4413f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f7e441430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e44135230 con 0x7f7e440a98f0 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e44135da0 con 0x7f7e4413f710 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e44135f30 con 0x7f7e441430c0 2024-03-25T18:27:18.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:18.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.723+0000 7f7e49bf4640 10 monclient(hunting): authenticate will time out at 7753.950387s 2024-03-25T18:27:18.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.730+0000 7f7e43fff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f7e441430c0 0x7f7e441db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:54702/0 (socket says 172.21.15.42:54702) 2024-03-25T18:27:18.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.730+0000 7f7e43fff640 1 -- 172.21.15.42:0/1038191826 learned_addr learned my addr 172.21.15.42:0/1038191826 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:18.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.730+0000 7f7e2bfff640 10 client.?.objecter ms_handle_connect 0x7f7e441430c0 2024-03-25T18:27:18.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.730+0000 7f7e2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.739 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.738+0000 7f7e2bfff640 10 client.?.objecter ms_handle_connect 0x7f7e440a98f0 2024-03-25T18:27:18.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.738+0000 7f7e2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.742+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211864976 0 0) 0x7f7e44135f30 con 0x7f7e441430c0 2024-03-25T18:27:18.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.742+0000 7f7e2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.742+0000 7f7e2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.742+0000 7f7e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.742+0000 7f7e2bfff640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:27:18.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.742+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e30004eb0 con 0x7f7e441430c0 2024-03-25T18:27:18.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 10 client.?.objecter ms_handle_connect 0x7f7e4413f710 2024-03-25T18:27:18.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418982650 0 0) 0x7f7e44135230 con 0x7f7e440a98f0 2024-03-25T18:27:18.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:27:18.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.749+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e44135f30 con 0x7f7e440a98f0 2024-03-25T18:27:18.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.759+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2054446699 0 0) 0x7f7e44135f30 con 0x7f7e440a98f0 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.759+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e44135230 con 0x7f7e440a98f0 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.759+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7e34001970 con 0x7f7e440a98f0 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.759+0000 7f7e2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.759+0000 7f7e2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.759+0000 7f7e2bfff640 10 monclient(hunting): dump: 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:18.762 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:18.762 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:18.762 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:18.762 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.759+0000 7f7e2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:18.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.761+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114672413 0 0) 0x7f7e44135da0 con 0x7f7e4413f710 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.761+0000 7f7e2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.761+0000 7f7e2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.761+0000 7f7e2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.761+0000 7f7e2bfff640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.761+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e44135f30 con 0x7f7e4413f710 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3664223048 0 0) 0x7f7e44135230 con 0x7f7e440a98f0 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 >> v1:172.21.15.42:6790/0 conn(0x7f7e441430c0 legacy=0x7f7e441db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 >> v1:172.21.15.120:6789/0 conn(0x7f7e4413f710 legacy=0x7f7e441dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 1 monclient: found mon.a 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e441360c0 con 0x7f7e440a98f0 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 10 monclient: _finish_auth 0 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:18.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 10 monclient: _renew_subs 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e441d5000 con 0x7f7e440a98f0 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 10 monclient: _renew_subs 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e441d41e0 con 0x7f7e440a98f0 2024-03-25T18:27:18.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.768+0000 7f7e49bf4640 1 librados: init done 2024-03-25T18:27:18.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.769+0000 7f7e49bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:18.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.769+0000 7f7e49bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:18.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.769+0000 7f7e49bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:18.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:18.769+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e44135090 con 0x7f7e440a98f0 2024-03-25T18:27:19.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.080+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e340047e0 con 0x7f7e440a98f0 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.080+0000 7f7e2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.080+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7e34004950 con 0x7f7e440a98f0 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.080+0000 7f7e2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.080+0000 7f7e2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.080+0000 7f7e2bfff640 10 monclient: dump: 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:19.082 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:19.083 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:19.083 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:19.083 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:19.083 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:19.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.080+0000 7f7e41ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:19.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.081+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f7e3401a960 con 0x7f7e440a98f0 2024-03-25T18:27:19.083 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.081+0000 7f7e42ffd640 10 monclient: get_auth_request con 0x7f7e3006de10 auth_method 0 2024-03-25T18:27:19.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.082+0000 7f7e2bfff640 10 client.4572.objecter ms_handle_connect 0x7f7e3006de10 2024-03-25T18:27:19.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f7e34054600 con 0x7f7e440a98f0 2024-03-25T18:27:19.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 10 client.4572.objecter ms_dispatch 0x7f7e44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:19.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:19.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:19.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-25T18:27:19.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e3404ff40 con 0x7f7e440a98f0 2024-03-25T18:27:19.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:19.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.088+0000 7f7e2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:19.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.338+0000 7f7e49bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.338+0000 7f7e49bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.338+0000 7f7e49bf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:19.340 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.338+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e441353c0 con 0x7f7e440a98f0 2024-03-25T18:27:19.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.369+0000 7f7e2bfff640 1 -- 172.21.15.42:0/1038191826 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f7e34052040 con 0x7f7e440a98f0 2024-03-25T18:27:19.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.369+0000 7f7e2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.369+0000 7f7e2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:19.371 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:19.371 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:19.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 10 librados: watch_flush enter 2024-03-25T18:27:19.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 10 librados: watch_flush exit 2024-03-25T18:27:19.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-25T18:27:19.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 10 client.4572.objecter successfully canceled tick 2024-03-25T18:27:19.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 >> v1:172.21.15.42:6800/3081750888 conn(0x7f7e3006de10 legacy=0x7f7e300702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 10 monclient: shutdown 2024-03-25T18:27:19.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:19.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 >> v1:172.21.15.42:6789/0 conn(0x7f7e440a98f0 legacy=0x7f7e441d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 shutdown_connections 2024-03-25T18:27:19.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.371+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 >> 172.21.15.42:0/1038191826 conn(0x7f7e440b7ea0 msgr2=0x7f7e4414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:19.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.372+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 shutdown_connections 2024-03-25T18:27:19.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.372+0000 7f7e49bf4640 1 -- 172.21.15.42:0/1038191826 wait complete. 2024-03-25T18:27:19.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:19.372+0000 7f7e49bf4640 1 librados: shutdown 2024-03-25T18:27:19.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:26.193 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:26.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.309+0000 7f0ffa73d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:26.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.309+0000 7f0ffa73d640 10 monclient: build_initial_monmap 2024-03-25T18:27:26.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.309+0000 7f0ffa73d640 10 monclient: monmap: 2024-03-25T18:27:26.310 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:26.310257+0000 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:26.310257+0000 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:26.311 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.309+0000 7f0ffa73d640 1 Processor -- start 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 1 -- start start 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient: init 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff40705b0 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff41402d0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff4149ae0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40718c0 con 0x7f0ff40705b0 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40714c0 con 0x7f0ff4149ae0 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4072b50 con 0x7f0ff41402d0 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:26.313 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.310+0000 7f0ffa73d640 10 monclient(hunting): authenticate will time out at 7761.537273s 2024-03-25T18:27:26.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.317+0000 7f0ff37fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f0ff41402d0 0x7f0ff4140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:59236/0 (socket says 172.21.15.42:59236) 2024-03-25T18:27:26.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.317+0000 7f0ff37fe640 1 -- 172.21.15.42:0/3976801645 learned_addr learned my addr 172.21.15.42:0/3976801645 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:26.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929806885 0 0) 0x7f0ff40714c0 con 0x7f0ff4149ae0 2024-03-25T18:27:26.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fdc003520 con 0x7f0ff4149ae0 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472050729 0 0) 0x7f0ff4072b50 con 0x7f0ff41402d0 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:27:26.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.334+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff40714c0 con 0x7f0ff41402d0 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3853410685 0 0) 0x7f0fdc003520 con 0x7f0ff4149ae0 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 >> v1:172.21.15.42:6790/0 conn(0x7f0ff41402d0 legacy=0x7f0ff4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 >> v1:172.21.15.42:6789/0 conn(0x7f0ff40705b0 legacy=0x7f0ff413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff413dd10 con 0x7f0ff4149ae0 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0fec001a10 con 0x7f0ff4149ae0 2024-03-25T18:27:26.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ffa73d640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ff2ffd640 10 monclient: dump: 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:26.350 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:26.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.347+0000 7f0ffa73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:26.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fec001b70 con 0x7f0ff4149ae0 2024-03-25T18:27:26.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ff2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:26.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ff2ffd640 1 -- 172.21.15.42:0/3976801645 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0fec001f80 con 0x7f0ff4149ae0 2024-03-25T18:27:26.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ff2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ff2ffd640 10 monclient: dump: 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:26.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ffa73d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:26.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.361+0000 7f0ffa73d640 4 set_mon_vals no callback set 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.362+0000 7f0ffa73d640 10 monclient: shutdown 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.362+0000 7f0ffa73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.362+0000 7f0ffa73d640 1 -- 172.21.15.42:0/3976801645 >> v1:172.21.15.120:6789/0 conn(0x7f0ff4149ae0 legacy=0x7f0ff414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.362+0000 7f0ffa73d640 1 -- 172.21.15.42:0/3976801645 shutdown_connections 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.362+0000 7f0ffa73d640 1 -- 172.21.15.42:0/3976801645 >> 172.21.15.42:0/3976801645 conn(0x7f0ff40b5d10 msgr2=0x7f0ff40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.362+0000 7f0ffa73d640 1 -- 172.21.15.42:0/3976801645 shutdown_connections 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.362+0000 7f0ffa73d640 1 -- 172.21.15.42:0/3976801645 wait complete. 2024-03-25T18:27:26.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 10 monclient: build_initial_monmap 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 10 monclient: monmap: 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 1 librados: starting msgr at 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 1 librados: starting objecter 2024-03-25T18:27:26.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 1 Processor -- start 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 1 -- start start 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 1 librados: setting wanted keys 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 1 librados: calling monclient init 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.363+0000 7f0ffa73d640 10 monclient: init 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff40705b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff41402d0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff4149ae0 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4137f00 con 0x7f0ff4149ae0 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff413fd20 con 0x7f0ff41402d0 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff413ff90 con 0x7f0ff40705b0 2024-03-25T18:27:26.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:26.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.364+0000 7f0ffa73d640 10 monclient(hunting): authenticate will time out at 7761.591009s 2024-03-25T18:27:26.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.375+0000 7f0ff37fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f0ff41402d0 0x7f0ff406bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:51806/0 (socket says 172.21.15.42:51806) 2024-03-25T18:27:26.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.375+0000 7f0ff37fe640 1 -- 172.21.15.42:0/1169344338 learned_addr learned my addr 172.21.15.42:0/1169344338 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:26.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.376+0000 7f0ff0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ff41402d0 2024-03-25T18:27:26.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.376+0000 7f0ff0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.379 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.378+0000 7f0ff0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ff40705b0 2024-03-25T18:27:26.379 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.378+0000 7f0ff0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.378+0000 7f0ff0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ff4149ae0 2024-03-25T18:27:26.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.378+0000 7f0ff0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.388+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014897033 0 0) 0x7f0ff413fd20 con 0x7f0ff41402d0 2024-03-25T18:27:26.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.388+0000 7f0ff0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.388+0000 7f0ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.388+0000 7f0ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.388+0000 7f0ff0ff9640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:27:26.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.388+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe0004eb0 con 0x7f0ff41402d0 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.398+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2041977238 0 0) 0x7f0fe0004eb0 con 0x7f0ff41402d0 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.398+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff413fd20 con 0x7f0ff41402d0 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.398+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0fe80019b0 con 0x7f0ff41402d0 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.398+0000 7f0ff0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.398+0000 7f0ff0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.398+0000 7f0ff0ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.400 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.398+0000 7f0ff0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.399+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490214404 0 0) 0x7f0ff413ff90 con 0x7f0ff40705b0 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.399+0000 7f0ff0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.399+0000 7f0ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.399+0000 7f0ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.399+0000 7f0ff0ff9640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:27:26.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.399+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe0004eb0 con 0x7f0ff40705b0 2024-03-25T18:27:26.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.404+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465648268 0 0) 0x7f0ff4137f00 con 0x7f0ff4149ae0 2024-03-25T18:27:26.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.404+0000 7f0ff0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.404+0000 7f0ff0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.404+0000 7f0ff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.404+0000 7f0ff0ff9640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:27:26.406 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.404+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff413ff90 con 0x7f0ff4149ae0 2024-03-25T18:27:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 966909777 0 0) 0x7f0ff413fd20 con 0x7f0ff41402d0 2024-03-25T18:27:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 >> v1:172.21.15.42:6790/0 conn(0x7f0ff40705b0 legacy=0x7f0ff413e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 >> v1:172.21.15.42:6789/0 conn(0x7f0ff4149ae0 legacy=0x7f0ff41e20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 1 monclient: found mon.b 2024-03-25T18:27:26.416 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4137040 con 0x7f0ff41402d0 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ff0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ffa73d640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ffa73d640 10 monclient: _renew_subs 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ffa73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.414+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff41378c0 con 0x7f0ff41402d0 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 10 monclient: _renew_subs 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff4148ee0 con 0x7f0ff41402d0 2024-03-25T18:27:26.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 1 librados: init done 2024-03-25T18:27:26.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:26.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.415+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff40627b0 con 0x7f0ff41402d0 2024-03-25T18:27:26.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe80047e0 con 0x7f0ff41402d0 2024-03-25T18:27:26.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:26.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0fe8004bf0 con 0x7f0ff41402d0 2024-03-25T18:27:26.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff0ff9640 10 monclient: dump: 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:26.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff27fc640 4 set_mon_vals no callback set 2024-03-25T18:27:26.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.688+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f0fe8004e10 con 0x7f0ff41402d0 2024-03-25T18:27:26.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.689+0000 7f0ff3fff640 10 monclient: get_auth_request con 0x7f0fe006de80 auth_method 0 2024-03-25T18:27:26.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.689+0000 7f0ff0ff9640 10 client.4585.objecter ms_handle_connect 0x7f0fe006de80 2024-03-25T18:27:26.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.693+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f0fe8054900 con 0x7f0ff41402d0 2024-03-25T18:27:26.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.693+0000 7f0ff0ff9640 10 client.4585.objecter ms_dispatch 0x7f0ff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:26.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.693+0000 7f0ff0ff9640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:26.694 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.693+0000 7f0ff0ff9640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:26.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.693+0000 7f0ff0ff9640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T18:27:26.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.695+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fe8050410 con 0x7f0ff41402d0 2024-03-25T18:27:26.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.695+0000 7f0ff0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.695+0000 7f0ff0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:26.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.953+0000 7f0ffa73d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:26.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.953+0000 7f0ffa73d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:26.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.953+0000 7f0ffa73d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:26.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.953+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ff4069f40 con 0x7f0ff41402d0 2024-03-25T18:27:26.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.973+0000 7f0ff0ff9640 1 -- 172.21.15.42:0/1169344338 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f0fe8052340 con 0x7f0ff41402d0 2024-03-25T18:27:26.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.974+0000 7f0ff0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:26.975 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.974+0000 7f0ff0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:26.976 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:26.976 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:26.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 10 librados: watch_flush enter 2024-03-25T18:27:26.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 10 librados: watch_flush exit 2024-03-25T18:27:26.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T18:27:26.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 10 client.4585.objecter successfully canceled tick 2024-03-25T18:27:26.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 >> v1:172.21.15.42:6800/3081750888 conn(0x7f0fe006de80 legacy=0x7f0fe0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 10 monclient: shutdown 2024-03-25T18:27:26.978 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:26.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 >> v1:172.21.15.120:6789/0 conn(0x7f0ff41402d0 legacy=0x7f0ff406bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 shutdown_connections 2024-03-25T18:27:26.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 >> 172.21.15.42:0/1169344338 conn(0x7f0ff40b5d10 msgr2=0x7f0ff40b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:26.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 shutdown_connections 2024-03-25T18:27:26.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 1 -- 172.21.15.42:0/1169344338 wait complete. 2024-03-25T18:27:26.979 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:26.976+0000 7f0ffa73d640 1 librados: shutdown 2024-03-25T18:27:26.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:27.788 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:10.957 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.796 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:10.965 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.825 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:10.994 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.853 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:10.974 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.856 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:11.026 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.857 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:10.977 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.867 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:11.036 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.892 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:11.062 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:27.893 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:11.013 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:33.797 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.911+0000 7f30ff257640 10 monclient: get_monmap_and_config 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.911+0000 7f30ff257640 10 monclient: build_initial_monmap 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.911+0000 7f30ff257640 10 monclient: monmap: 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:33.912634+0000 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:33.912634+0000 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:33.913 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:33.914 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.912+0000 7f30ff257640 1 Processor -- start 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.912+0000 7f30ff257640 1 -- start start 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.912+0000 7f30ff257640 10 monclient: init 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.912+0000 7f30ff257640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.912+0000 7f30ff257640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 10 monclient(hunting): picked mon.noname-a con 0x7f30f80a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 10 monclient(hunting): picked mon.noname-b con 0x7f30f813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 10 monclient(hunting): picked mon.noname-c con 0x7f30f81430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f8142520 con 0x7f30f80a98f0 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f80b28a0 con 0x7f30f81430c0 2024-03-25T18:27:33.915 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f80b2a30 con 0x7f30f813f710 2024-03-25T18:27:33.916 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:33.916 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.913+0000 7f30ff257640 10 monclient(hunting): authenticate will time out at 7769.139642s 2024-03-25T18:27:33.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.920+0000 7f30f7fff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f30f813f710 0x7f30f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:42608/0 (socket says 172.21.15.42:42608) 2024-03-25T18:27:33.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.921+0000 7f30f7fff640 1 -- 172.21.15.42:0/696206574 learned_addr learned my addr 172.21.15.42:0/696206574 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:33.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.934+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467973494 0 0) 0x7f30f80b28a0 con 0x7f30f81430c0 2024-03-25T18:27:33.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.934+0000 7f30f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:33.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.934+0000 7f30f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:33.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.934+0000 7f30f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:33.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.934+0000 7f30f77fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:27:33.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.934+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e8003520 con 0x7f30f81430c0 2024-03-25T18:27:33.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.936+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289512074 0 0) 0x7f30f80b2a30 con 0x7f30f813f710 2024-03-25T18:27:33.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.936+0000 7f30f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:33.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.936+0000 7f30f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:33.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.936+0000 7f30f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:33.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.936+0000 7f30f77fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:27:33.938 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.936+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f80b28a0 con 0x7f30f813f710 2024-03-25T18:27:33.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.955+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201876069 0 0) 0x7f30f8142520 con 0x7f30f80a98f0 2024-03-25T18:27:33.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.955+0000 7f30f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:33.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.955+0000 7f30f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:33.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.955+0000 7f30f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:33.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.955+0000 7f30f77fe640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:27:33.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.955+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f80b2a30 con 0x7f30f80a98f0 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4268125273 0 0) 0x7f30f80b28a0 con 0x7f30f813f710 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 >> v1:172.21.15.120:6789/0 conn(0x7f30f81430c0 legacy=0x7f30f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 >> v1:172.21.15.42:6789/0 conn(0x7f30f80a98f0 legacy=0x7f30f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 1 monclient: found mon.noname-b 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:27:33.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f80b44c0 con 0x7f30f813f710 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f30e4001670 con 0x7f30f813f710 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30f77fe640 10 monclient: dump: 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:33.960 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:33.961 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:33.961 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:33.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30ff257640 5 monclient: authenticate success, global_id 4583 2024-03-25T18:27:33.961 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.957+0000 7f30ff257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:33.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e40017d0 con 0x7f30f813f710 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30f77fe640 1 -- 172.21.15.42:0/696206574 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f30e4001ba0 con 0x7f30f813f710 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30ff257640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30ff257640 4 set_mon_vals no callback set 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.983+0000 7f30f77fe640 10 monclient: dump: 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:33.985 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:33.986 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:33.986 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:33.986 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.984+0000 7f30ff257640 10 monclient: shutdown 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.984+0000 7f30ff257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.984+0000 7f30ff257640 1 -- 172.21.15.42:0/696206574 >> v1:172.21.15.42:6790/0 conn(0x7f30f813f710 legacy=0x7f30f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.984+0000 7f30ff257640 1 -- 172.21.15.42:0/696206574 shutdown_connections 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.984+0000 7f30ff257640 1 -- 172.21.15.42:0/696206574 >> 172.21.15.42:0/696206574 conn(0x7f30f80b7ea0 msgr2=0x7f30f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.984+0000 7f30ff257640 1 -- 172.21.15.42:0/696206574 shutdown_connections 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.984+0000 7f30ff257640 1 -- 172.21.15.42:0/696206574 wait complete. 2024-03-25T18:27:33.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 10 monclient: build_initial_monmap 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 10 monclient: monmap: 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 1 librados: starting msgr at 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 1 librados: starting objecter 2024-03-25T18:27:33.988 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 1 Processor -- start 2024-03-25T18:27:33.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 1 -- start start 2024-03-25T18:27:33.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 1 librados: setting wanted keys 2024-03-25T18:27:33.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 1 librados: calling monclient init 2024-03-25T18:27:33.989 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.985+0000 7f30ff257640 10 monclient: init 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 10 monclient(hunting): picked mon.noname-c con 0x7f30f80a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 10 monclient(hunting): picked mon.noname-a con 0x7f30f813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 10 monclient(hunting): picked mon.noname-b con 0x7f30f81430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f80ab9c0 con 0x7f30f813f710 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f80abca0 con 0x7f30f81430c0 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f8137e80 con 0x7f30f80a98f0 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:33.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.986+0000 7f30ff257640 10 monclient(hunting): authenticate will time out at 7769.213410s 2024-03-25T18:27:33.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.990+0000 7f30fd7f4640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f30f81430c0 0x7f30f81e2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:52160/0 (socket says 172.21.15.42:52160) 2024-03-25T18:27:33.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.990+0000 7f30fd7f4640 1 -- 172.21.15.42:0/4093028501 learned_addr learned my addr 172.21.15.42:0/4093028501 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:33.992 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.991+0000 7f30f57fa640 10 client.?.objecter ms_handle_connect 0x7f30f81430c0 2024-03-25T18:27:33.992 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.991+0000 7f30f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:33.998 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.997+0000 7f30f57fa640 10 client.?.objecter ms_handle_connect 0x7f30f80a98f0 2024-03-25T18:27:33.998 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:33.997+0000 7f30f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.004+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584833611 0 0) 0x7f30f80abca0 con 0x7f30f81430c0 2024-03-25T18:27:34.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.004+0000 7f30f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.004+0000 7f30f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.004+0000 7f30f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.004+0000 7f30f57fa640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:27:34.005 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.004+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc004eb0 con 0x7f30f81430c0 2024-03-25T18:27:34.007 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.006+0000 7f30f57fa640 10 client.?.objecter ms_handle_connect 0x7f30f813f710 2024-03-25T18:27:34.007 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.006+0000 7f30f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.008 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.007+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731593695 0 0) 0x7f30f8137e80 con 0x7f30f80a98f0 2024-03-25T18:27:34.008 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.007+0000 7f30f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.008 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.007+0000 7f30f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.008 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.007+0000 7f30f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.008 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.007+0000 7f30f57fa640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:27:34.008 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.007+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f80abca0 con 0x7f30f80a98f0 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.015+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 265623902 0 0) 0x7f30dc004eb0 con 0x7f30f81430c0 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.015+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30f8137e80 con 0x7f30f81430c0 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.015+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f30f0001c30 con 0x7f30f81430c0 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.015+0000 7f30f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.015+0000 7f30f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.015+0000 7f30f57fa640 10 monclient(hunting): dump: 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:34.017 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:34.018 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.018 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:34.018 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:34.018 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:34.018 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:34.018 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:34.018 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.015+0000 7f30f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:34.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.025+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415521879 0 0) 0x7f30f80ab9c0 con 0x7f30f813f710 2024-03-25T18:27:34.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.025+0000 7f30f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.025+0000 7f30f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.025+0000 7f30f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.025+0000 7f30f57fa640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:27:34.026 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.025+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc004eb0 con 0x7f30f813f710 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.026+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 941566376 0 0) 0x7f30f80abca0 con 0x7f30f80a98f0 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.026+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30f80ab9c0 con 0x7f30f80a98f0 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.026+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f30ec0018f0 con 0x7f30f80a98f0 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.027+0000 7f30f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.027+0000 7f30f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.027+0000 7f30f57fa640 10 monclient(hunting): dump: 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:34.028 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:34.029 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:34.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3834808480 0 0) 0x7f30f80ab9c0 con 0x7f30f80a98f0 2024-03-25T18:27:34.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 >> v1:172.21.15.120:6789/0 conn(0x7f30f81430c0 legacy=0x7f30f81e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 >> v1:172.21.15.42:6789/0 conn(0x7f30f813f710 legacy=0x7f30f8133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:34.038 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 1 monclient: found mon.c 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f81d8150 con 0x7f30f80a98f0 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 10 monclient: _renew_subs 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f8138eb0 con 0x7f30f80a98f0 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 10 monclient: _renew_subs 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:34.039 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f8137b20 con 0x7f30f80a98f0 2024-03-25T18:27:34.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.036+0000 7f30ff257640 1 librados: init done 2024-03-25T18:27:34.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.038+0000 7f30ff257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.038+0000 7f30ff257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.038+0000 7f30ff257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:34.040 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.038+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f80627b0 con 0x7f30f80a98f0 2024-03-25T18:27:34.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.374+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30ec003610 con 0x7f30f80a98f0 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.374+0000 7f30f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.374+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f30ec0039e0 con 0x7f30f80a98f0 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.374+0000 7f30f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.374+0000 7f30f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.374+0000 7f30f57fa640 10 monclient: dump: 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.376 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:34.377 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:34.377 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:34.377 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:34.377 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:34.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.374+0000 7f30f6ffd640 4 set_mon_vals no callback set 2024-03-25T18:27:34.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.375+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f30ec003c00 con 0x7f30f80a98f0 2024-03-25T18:27:34.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.375+0000 7f30f7fff640 10 monclient: get_auth_request con 0x7f30dc06df50 auth_method 0 2024-03-25T18:27:34.378 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.375+0000 7f30f57fa640 10 client.4586.objecter ms_handle_connect 0x7f30dc06df50 2024-03-25T18:27:34.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f30ec0535b0 con 0x7f30f80a98f0 2024-03-25T18:27:34.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 10 client.4586.objecter ms_dispatch 0x7f30f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:34.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:34.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:34.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T18:27:34.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30ec04f0c0 con 0x7f30f80a98f0 2024-03-25T18:27:34.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.381+0000 7f30f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:34.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.641+0000 7f30ff257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:34.642 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.641+0000 7f30ff257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:34.643 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.641+0000 7f30ff257640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:34.643 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.641+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30f80b71a0 con 0x7f30f80a98f0 2024-03-25T18:27:34.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.751+0000 7f30f57fa640 1 -- 172.21.15.42:0/4093028501 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f30ec050ff0 con 0x7f30f80a98f0 2024-03-25T18:27:34.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.751+0000 7f30f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:34.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.751+0000 7f30f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:34.753 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:34.753 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:34.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 10 librados: watch_flush enter 2024-03-25T18:27:34.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 10 librados: watch_flush exit 2024-03-25T18:27:34.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T18:27:34.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 10 client.4586.objecter successfully canceled tick 2024-03-25T18:27:34.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 >> v1:172.21.15.42:6800/3081750888 conn(0x7f30dc06df50 legacy=0x7f30dc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 10 monclient: shutdown 2024-03-25T18:27:34.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:34.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.753+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 >> v1:172.21.15.42:6790/0 conn(0x7f30f80a98f0 legacy=0x7f30f81d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.754+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 shutdown_connections 2024-03-25T18:27:34.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.754+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 >> 172.21.15.42:0/4093028501 conn(0x7f30f80b7ea0 msgr2=0x7f30f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:34.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.754+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 shutdown_connections 2024-03-25T18:27:34.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.754+0000 7f30ff257640 1 -- 172.21.15.42:0/4093028501 wait complete. 2024-03-25T18:27:34.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:34.754+0000 7f30ff257640 1 librados: shutdown 2024-03-25T18:27:34.765 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:41.572 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:41.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.685+0000 7f097dd0e640 10 monclient: get_monmap_and_config 2024-03-25T18:27:41.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.686+0000 7f097dd0e640 10 monclient: build_initial_monmap 2024-03-25T18:27:41.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.686+0000 7f097dd0e640 10 monclient: monmap: 2024-03-25T18:27:41.686 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:41.686 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:41.686079+0000 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:41.686079+0000 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:41.687 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:41.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 1 Processor -- start 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 1 -- start start 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient: init 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient(hunting): picked mon.noname-c con 0x7f09780a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient(hunting): picked mon.noname-a con 0x7f097813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient(hunting): picked mon.noname-b con 0x7f09781430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0978142520 con 0x7f097813f710 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780b28a0 con 0x7f09780a98f0 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780b2a30 con 0x7f09781430c0 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:41.689 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.687+0000 7f097dd0e640 10 monclient(hunting): authenticate will time out at 7776.913526s 2024-03-25T18:27:41.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.690+0000 7f09777fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f09780a98f0 0x7f09780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:47146/0 (socket says 172.21.15.42:47146) 2024-03-25T18:27:41.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.690+0000 7f09777fe640 1 -- 172.21.15.42:0/674943655 learned_addr learned my addr 172.21.15.42:0/674943655 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:41.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.701+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814879041 0 0) 0x7f09780b28a0 con 0x7f09780a98f0 2024-03-25T18:27:41.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.701+0000 7f09767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.701+0000 7f09767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.701+0000 7f09767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.701+0000 7f09767fc640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:27:41.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.701+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0960003520 con 0x7f09780a98f0 2024-03-25T18:27:41.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2311611093 0 0) 0x7f0960003520 con 0x7f09780a98f0 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 >> v1:172.21.15.42:6790/0 conn(0x7f09781430c0 legacy=0x7f097814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 >> v1:172.21.15.42:6789/0 conn(0x7f097813f710 legacy=0x7f09781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 1 monclient: found mon.noname-c 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780b44c0 con 0x7f09780a98f0 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0968001690 con 0x7f09780a98f0 2024-03-25T18:27:41.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f09767fc640 10 monclient: dump: 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:41.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f097dd0e640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:27:41.715 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.712+0000 7f097dd0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:41.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09680017f0 con 0x7f09780a98f0 2024-03-25T18:27:41.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f09767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:41.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f09767fc640 1 -- 172.21.15.42:0/674943655 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0968001c00 con 0x7f09780a98f0 2024-03-25T18:27:41.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f09767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f09767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:41.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f09767fc640 10 monclient: dump: 2024-03-25T18:27:41.718 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f097dd0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:41.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.718+0000 7f097dd0e640 4 set_mon_vals no callback set 2024-03-25T18:27:41.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.719+0000 7f097dd0e640 10 monclient: shutdown 2024-03-25T18:27:41.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.719+0000 7f097dd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:41.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.719+0000 7f097dd0e640 1 -- 172.21.15.42:0/674943655 >> v1:172.21.15.120:6789/0 conn(0x7f09780a98f0 legacy=0x7f09780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.719+0000 7f097dd0e640 1 -- 172.21.15.42:0/674943655 shutdown_connections 2024-03-25T18:27:41.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.719+0000 7f097dd0e640 1 -- 172.21.15.42:0/674943655 >> 172.21.15.42:0/674943655 conn(0x7f09780b7ea0 msgr2=0x7f09780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:41.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.719+0000 7f097dd0e640 1 -- 172.21.15.42:0/674943655 shutdown_connections 2024-03-25T18:27:41.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.719+0000 7f097dd0e640 1 -- 172.21.15.42:0/674943655 wait complete. 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.720+0000 7f097dd0e640 10 monclient: build_initial_monmap 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.720+0000 7f097dd0e640 10 monclient: monmap: 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.720+0000 7f097dd0e640 1 librados: starting msgr at 2024-03-25T18:27:41.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.720+0000 7f097dd0e640 1 librados: starting objecter 2024-03-25T18:27:41.722 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.720+0000 7f097dd0e640 1 Processor -- start 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 1 -- start start 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 1 librados: setting wanted keys 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 1 librados: calling monclient init 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient: init 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient(hunting): picked mon.noname-a con 0x7f09780a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient(hunting): picked mon.noname-c con 0x7f097813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient(hunting): picked mon.noname-b con 0x7f09781430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09781db7d0 con 0x7f09780a98f0 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780b1420 con 0x7f09781430c0 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780b15b0 con 0x7f097813f710 2024-03-25T18:27:41.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:41.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.721+0000 7f097dd0e640 10 monclient(hunting): authenticate will time out at 7776.947494s 2024-03-25T18:27:41.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.730+0000 7f0976ffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f097813f710 0x7f09780b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:41532/0 (socket says 172.21.15.42:41532) 2024-03-25T18:27:41.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.730+0000 7f0976ffd640 1 -- 172.21.15.42:0/2457071400 learned_addr learned my addr 172.21.15.42:0/2457071400 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:41.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.735+0000 7f095ffff640 10 client.?.objecter ms_handle_connect 0x7f097813f710 2024-03-25T18:27:41.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.735+0000 7f095ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:41.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.738+0000 7f095ffff640 10 client.?.objecter ms_handle_connect 0x7f09781430c0 2024-03-25T18:27:41.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.738+0000 7f095ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:41.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.742+0000 7f095ffff640 10 client.?.objecter ms_handle_connect 0x7f09780a98f0 2024-03-25T18:27:41.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.742+0000 7f095ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:41.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.744+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668523128 0 0) 0x7f09780b15b0 con 0x7f097813f710 2024-03-25T18:27:41.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.744+0000 7f095ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.744+0000 7f095ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.744+0000 7f095ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.744+0000 7f095ffff640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:27:41.745 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.744+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0964004eb0 con 0x7f097813f710 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.747+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922893995 0 0) 0x7f09780b1420 con 0x7f09781430c0 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.747+0000 7f095ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.747+0000 7f095ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.747+0000 7f095ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.748+0000 7f095ffff640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.748+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09780b15b0 con 0x7f09781430c0 2024-03-25T18:27:41.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.759+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412858424 0 0) 0x7f09781db7d0 con 0x7f09780a98f0 2024-03-25T18:27:41.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.759+0000 7f095ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.759+0000 7f095ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.759+0000 7f095ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.759+0000 7f095ffff640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:27:41.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.759+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09780b1420 con 0x7f09780a98f0 2024-03-25T18:27:41.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.772+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4219137768 0 0) 0x7f09780b15b0 con 0x7f09781430c0 2024-03-25T18:27:41.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.772+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09781db7d0 con 0x7f09781430c0 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.772+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0970001b10 con 0x7f09781430c0 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.772+0000 7f095ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.772+0000 7f095ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.772+0000 7f095ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:41.773 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:41.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.772+0000 7f095ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.777+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2348937855 0 0) 0x7f0964004eb0 con 0x7f097813f710 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.777+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09780b15b0 con 0x7f097813f710 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.777+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f096c001a30 con 0x7f097813f710 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.777+0000 7f095ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.777+0000 7f095ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.777+0000 7f095ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.778 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.779 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.779 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:41.779 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:41.779 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:41.779 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:41.779 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.780+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2076483851 0 0) 0x7f09780b1420 con 0x7f09780a98f0 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.780+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0964004eb0 con 0x7f09780a98f0 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.780+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0968001b00 con 0x7f09780a98f0 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.780+0000 7f095ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.780+0000 7f095ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.780+0000 7f095ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3797505189 0 0) 0x7f09781db7d0 con 0x7f09781430c0 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 >> v1:172.21.15.42:6790/0 conn(0x7f097813f710 legacy=0x7f09780b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 >> v1:172.21.15.42:6789/0 conn(0x7f09780a98f0 legacy=0x7f09781daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 1 monclient: found mon.b 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780b1740 con 0x7f09781430c0 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 10 monclient: _finish_auth 0 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:41.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f095ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 10 monclient: _renew_subs 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09781db1a0 con 0x7f09781430c0 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 10 client.4597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 10 monclient: _renew_subs 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09780b08f0 con 0x7f09781430c0 2024-03-25T18:27:41.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.788+0000 7f097dd0e640 1 librados: init done 2024-03-25T18:27:41.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.789+0000 7f097dd0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:41.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.789+0000 7f097dd0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:41.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.789+0000 7f097dd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:41.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:41.789+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09780627b0 con 0x7f09781430c0 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.118+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0970002000 con 0x7f09781430c0 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.118+0000 7f095ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.118+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0970003b10 con 0x7f09781430c0 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.118+0000 7f095ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.118+0000 7f095ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.118+0000 7f095ffff640 10 monclient: dump: 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:42.119 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.118+0000 7f0975ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:42.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.119+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f0970003d30 con 0x7f09781430c0 2024-03-25T18:27:42.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.119+0000 7f09777fe640 10 monclient: get_auth_request con 0x7f096406e110 auth_method 0 2024-03-25T18:27:42.121 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.120+0000 7f095ffff640 10 client.4597.objecter ms_handle_connect 0x7f096406e110 2024-03-25T18:27:42.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.124+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f0970053750 con 0x7f09781430c0 2024-03-25T18:27:42.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.124+0000 7f095ffff640 10 client.4597.objecter ms_dispatch 0x7f0978000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:42.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.124+0000 7f095ffff640 3 client.4597.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:42.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.124+0000 7f095ffff640 3 client.4597.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:42.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.124+0000 7f095ffff640 20 client.4597.objecter dump_active .. 0 homeless 2024-03-25T18:27:42.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.125+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f097004f260 con 0x7f09781430c0 2024-03-25T18:27:42.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.125+0000 7f095ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:42.126 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.125+0000 7f095ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:42.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.370+0000 7f097dd0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.370+0000 7f097dd0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.370+0000 7f097dd0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:27:42.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.370+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09780abae0 con 0x7f09781430c0 2024-03-25T18:27:42.473 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.471+0000 7f095ffff640 1 -- 172.21.15.42:0/2457071400 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f0970051190 con 0x7f09781430c0 2024-03-25T18:27:42.473 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.472+0000 7f095ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.473 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.472+0000 7f095ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:42.473 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:42.474 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 10 librados: watch_flush enter 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 10 librados: watch_flush exit 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 20 client.4597.objecter shutdown clearing up homeless session... 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 10 client.4597.objecter successfully canceled tick 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 >> v1:172.21.15.42:6800/3081750888 conn(0x7f096406e110 legacy=0x7f09640705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 10 monclient: shutdown 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 >> v1:172.21.15.120:6789/0 conn(0x7f09781430c0 legacy=0x7f09781d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 shutdown_connections 2024-03-25T18:27:42.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.474+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 >> 172.21.15.42:0/2457071400 conn(0x7f09780b7ea0 msgr2=0x7f097813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:42.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.475+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 shutdown_connections 2024-03-25T18:27:42.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.475+0000 7f097dd0e640 1 -- 172.21.15.42:0/2457071400 wait complete. 2024-03-25T18:27:42.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:42.475+0000 7f097dd0e640 1 librados: shutdown 2024-03-25T18:27:42.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:49.296 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:49.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.413+0000 7f3e5fb48640 10 monclient: get_monmap_and_config 2024-03-25T18:27:49.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.413+0000 7f3e5fb48640 10 monclient: build_initial_monmap 2024-03-25T18:27:49.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.413+0000 7f3e5fb48640 10 monclient: monmap: 2024-03-25T18:27:49.418 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:49.418 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:49.418 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:49.414673+0000 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:49.414673+0000 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.414+0000 7f3e5fb48640 1 Processor -- start 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.414+0000 7f3e5fb48640 1 -- start start 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.414+0000 7f3e5fb48640 10 monclient: init 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 10 monclient(hunting): picked mon.noname-b con 0x7f3e580a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:49.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 10 monclient(hunting): picked mon.noname-c con 0x7f3e5813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:49.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 10 monclient(hunting): picked mon.noname-a con 0x7f3e581430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:49.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e58142520 con 0x7f3e581430c0 2024-03-25T18:27:49.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580b28a0 con 0x7f3e5813f710 2024-03-25T18:27:49.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580b2a30 con 0x7f3e580a98f0 2024-03-25T18:27:49.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:49.420 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.415+0000 7f3e5fb48640 10 monclient(hunting): authenticate will time out at 7784.642067s 2024-03-25T18:27:49.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.423+0000 7f3e5d0e3640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3e5813f710 0x7f3e581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:57780/0 (socket says 172.21.15.42:57780) 2024-03-25T18:27:49.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.423+0000 7f3e5d0e3640 1 -- 172.21.15.42:0/4093728867 learned_addr learned my addr 172.21.15.42:0/4093728867 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:49.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.438+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287835542 0 0) 0x7f3e580b28a0 con 0x7f3e5813f710 2024-03-25T18:27:49.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.438+0000 7f3e5c8e2640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.438+0000 7f3e5c8e2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.438+0000 7f3e5c8e2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.438+0000 7f3e5c8e2640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:27:49.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.438+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e44003520 con 0x7f3e5813f710 2024-03-25T18:27:49.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.444+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038318715 0 0) 0x7f3e580b2a30 con 0x7f3e580a98f0 2024-03-25T18:27:49.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.444+0000 7f3e5c8e2640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.444+0000 7f3e5c8e2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.444+0000 7f3e5c8e2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.444+0000 7f3e5c8e2640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:27:49.445 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.444+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e580b28a0 con 0x7f3e580a98f0 2024-03-25T18:27:49.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 779128996 0 0) 0x7f3e44003520 con 0x7f3e5813f710 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 >> v1:172.21.15.42:6790/0 conn(0x7f3e580a98f0 legacy=0x7f3e580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 >> v1:172.21.15.42:6789/0 conn(0x7f3e581430c0 legacy=0x7f3e5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 10 monclient: _finish_hunting 0 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 1 monclient: found mon.noname-c 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580b44c0 con 0x7f3e5813f710 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 10 monclient: _finish_auth 0 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 10 monclient: _check_auth_tickets 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3e48001690 con 0x7f3e5813f710 2024-03-25T18:27:49.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5c8e2640 10 monclient: dump: 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5fb48640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:27:49.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.449+0000 7f3e5fb48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e480017f0 con 0x7f3e5813f710 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5c8e2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5c8e2640 1 -- 172.21.15.42:0/4093728867 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3e48001c00 con 0x7f3e5813f710 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5c8e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5c8e2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5c8e2640 10 monclient: dump: 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.459 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.460 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:49.460 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:49.460 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:49.460 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:49.460 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5fb48640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:49.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.457+0000 7f3e5fb48640 4 set_mon_vals no callback set 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.458+0000 7f3e5fb48640 10 monclient: shutdown 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.458+0000 7f3e5fb48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.458+0000 7f3e5fb48640 1 -- 172.21.15.42:0/4093728867 >> v1:172.21.15.120:6789/0 conn(0x7f3e5813f710 legacy=0x7f3e581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.458+0000 7f3e5fb48640 1 -- 172.21.15.42:0/4093728867 shutdown_connections 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.458+0000 7f3e5fb48640 1 -- 172.21.15.42:0/4093728867 >> 172.21.15.42:0/4093728867 conn(0x7f3e580b7ea0 msgr2=0x7f3e580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.458+0000 7f3e5fb48640 1 -- 172.21.15.42:0/4093728867 shutdown_connections 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.458+0000 7f3e5fb48640 1 -- 172.21.15.42:0/4093728867 wait complete. 2024-03-25T18:27:49.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 10 monclient: build_initial_monmap 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 10 monclient: monmap: 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 1 librados: starting msgr at 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 1 librados: starting objecter 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 1 Processor -- start 2024-03-25T18:27:49.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 1 -- start start 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 1 librados: setting wanted keys 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 1 librados: calling monclient init 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.459+0000 7f3e5fb48640 10 monclient: init 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 10 monclient(hunting): picked mon.noname-c con 0x7f3e580a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 10 monclient(hunting): picked mon.noname-a con 0x7f3e5813f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 10 monclient(hunting): picked mon.noname-b con 0x7f3e581430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e581d77a0 con 0x7f3e5813f710 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e58138e40 con 0x7f3e581430c0 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580aba30 con 0x7f3e580a98f0 2024-03-25T18:27:49.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:49.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.460+0000 7f3e5fb48640 10 monclient(hunting): authenticate will time out at 7784.687004s 2024-03-25T18:27:49.473 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.472+0000 7f3e5d8e4640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f3e580a98f0 0x7f3e58138730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:41810/0 (socket says 172.21.15.42:41810) 2024-03-25T18:27:49.473 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.472+0000 7f3e5d8e4640 1 -- 172.21.15.42:0/1984063619 learned_addr learned my addr 172.21.15.42:0/1984063619 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:49.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.473+0000 7f3e4e7fc640 10 client.?.objecter ms_handle_connect 0x7f3e580a98f0 2024-03-25T18:27:49.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.473+0000 7f3e4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:49.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.475+0000 7f3e4e7fc640 10 client.?.objecter ms_handle_connect 0x7f3e581430c0 2024-03-25T18:27:49.476 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.475+0000 7f3e4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:49.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.477+0000 7f3e4e7fc640 10 client.?.objecter ms_handle_connect 0x7f3e5813f710 2024-03-25T18:27:49.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.477+0000 7f3e4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:49.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.490+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754464476 0 0) 0x7f3e580aba30 con 0x7f3e580a98f0 2024-03-25T18:27:49.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.490+0000 7f3e4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.490+0000 7f3e4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.490+0000 7f3e4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.490+0000 7f3e4e7fc640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:27:49.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.490+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e3c004eb0 con 0x7f3e580a98f0 2024-03-25T18:27:49.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.491+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724487991 0 0) 0x7f3e58138e40 con 0x7f3e581430c0 2024-03-25T18:27:49.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.491+0000 7f3e4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.491+0000 7f3e4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.491+0000 7f3e4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.491+0000 7f3e4e7fc640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:27:49.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.491+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e580aba30 con 0x7f3e581430c0 2024-03-25T18:27:49.501 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.500+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615443383 0 0) 0x7f3e581d77a0 con 0x7f3e5813f710 2024-03-25T18:27:49.501 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.500+0000 7f3e4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.501 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.500+0000 7f3e4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.500+0000 7f3e4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.500+0000 7f3e4e7fc640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:27:49.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.500+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e58138e40 con 0x7f3e5813f710 2024-03-25T18:27:49.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.503+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2535309889 0 0) 0x7f3e3c004eb0 con 0x7f3e580a98f0 2024-03-25T18:27:49.504 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.503+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e581d77a0 con 0x7f3e580a98f0 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3e54001ab0 con 0x7f3e580a98f0 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:49.514 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3942627321 0 0) 0x7f3e580aba30 con 0x7f3e581430c0 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e3c004eb0 con 0x7f3e581430c0 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3e50001b00 con 0x7f3e581430c0 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:49.515 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462557368 0 0) 0x7f3e58138e40 con 0x7f3e5813f710 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e580aba30 con 0x7f3e5813f710 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3e48001b00 con 0x7f3e5813f710 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.512+0000 7f3e4e7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:49.516 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:49.517 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.517 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.517 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.517 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:49.517 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:49.517 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:49.517 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:49.518 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2084117864 0 0) 0x7f3e580aba30 con 0x7f3e5813f710 2024-03-25T18:27:49.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 >> v1:172.21.15.42:6790/0 conn(0x7f3e580a98f0 legacy=0x7f3e58138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 >> v1:172.21.15.120:6789/0 conn(0x7f3e581430c0 legacy=0x7f3e58134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 1 monclient: found mon.a 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e581d8340 con 0x7f3e5813f710 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 10 monclient: _renew_subs 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e58134820 con 0x7f3e5813f710 2024-03-25T18:27:49.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:49.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 10 monclient: _renew_subs 2024-03-25T18:27:49.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:49.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e581380c0 con 0x7f3e5813f710 2024-03-25T18:27:49.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.523+0000 7f3e5fb48640 1 librados: init done 2024-03-25T18:27:49.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.524+0000 7f3e5fb48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:49.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.524+0000 7f3e5fb48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:49.526 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.524+0000 7f3e5fb48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:49.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.524+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e580627b0 con 0x7f3e5813f710 2024-03-25T18:27:49.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.840+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e48004a90 con 0x7f3e5813f710 2024-03-25T18:27:49.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.840+0000 7f3e4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:49.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.840+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3e48001250 con 0x7f3e5813f710 2024-03-25T18:27:49.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.840+0000 7f3e4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.840+0000 7f3e4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:49.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.840+0000 7f3e4e7fc640 10 monclient: dump: 2024-03-25T18:27:49.842 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:49.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.840+0000 7f3e4f7fe640 4 set_mon_vals no callback set 2024-03-25T18:27:49.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.841+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f3e4801ac10 con 0x7f3e5813f710 2024-03-25T18:27:49.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.842+0000 7f3e5d8e4640 10 monclient: get_auth_request con 0x7f3e3c06e130 auth_method 0 2024-03-25T18:27:49.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.842+0000 7f3e4e7fc640 10 client.4596.objecter ms_handle_connect 0x7f3e3c06e130 2024-03-25T18:27:49.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f3e48054840 con 0x7f3e5813f710 2024-03-25T18:27:49.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 10 client.4596.objecter ms_dispatch 0x7f3e58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:49.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:49.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:49.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T18:27:49.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e48050180 con 0x7f3e5813f710 2024-03-25T18:27:49.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:49.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:49.848+0000 7f3e4e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:50.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.077+0000 7f3e5fb48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:50.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.077+0000 7f3e5fb48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:50.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.077+0000 7f3e5fb48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:27:50.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.077+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e5814e880 con 0x7f3e5813f710 2024-03-25T18:27:50.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.104+0000 7f3e4e7fc640 1 -- 172.21.15.42:0/1984063619 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f3e48052280 con 0x7f3e5813f710 2024-03-25T18:27:50.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.104+0000 7f3e4e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:50.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.104+0000 7f3e4e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:50.106 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:50.106 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:50.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 10 librados: watch_flush enter 2024-03-25T18:27:50.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 10 librados: watch_flush exit 2024-03-25T18:27:50.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T18:27:50.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 10 client.4596.objecter successfully canceled tick 2024-03-25T18:27:50.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 >> v1:172.21.15.42:6800/3081750888 conn(0x7f3e3c06e130 legacy=0x7f3e3c0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:50.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 10 monclient: shutdown 2024-03-25T18:27:50.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:50.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 >> v1:172.21.15.42:6789/0 conn(0x7f3e5813f710 legacy=0x7f3e581d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:50.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 shutdown_connections 2024-03-25T18:27:50.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.106+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 >> 172.21.15.42:0/1984063619 conn(0x7f3e580b7ea0 msgr2=0x7f3e58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:50.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.107+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 shutdown_connections 2024-03-25T18:27:50.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.107+0000 7f3e5fb48640 1 -- 172.21.15.42:0/1984063619 wait complete. 2024-03-25T18:27:50.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:50.107+0000 7f3e5fb48640 1 librados: shutdown 2024-03-25T18:27:50.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:52.908 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:36.077 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:52.918 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:36.088 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:52.953 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:36.122 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:52.968 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:36.089 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:52.975 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:36.144 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:52.976 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:36.097 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:52.982 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:36.151 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:53.007 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:36.127 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:53.030 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:36.199 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:56.929 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.042+0000 7ff9fc58f640 10 monclient: get_monmap_and_config 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.042+0000 7ff9fc58f640 10 monclient: build_initial_monmap 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.042+0000 7ff9fc58f640 10 monclient: monmap: 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:27:57.043806+0000 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:27:57.043806+0000 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:57.044 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:27:57.045 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:27:57.045 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.043+0000 7ff9fc58f640 1 Processor -- start 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.043+0000 7ff9fc58f640 1 -- start start 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.043+0000 7ff9fc58f640 10 monclient: init 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:57.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f41430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:57.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f4142520 con 0x7ff9f413f710 2024-03-25T18:27:57.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f40b28a0 con 0x7ff9f41430c0 2024-03-25T18:27:57.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f40b2a30 con 0x7ff9f40a98f0 2024-03-25T18:27:57.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:57.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.044+0000 7ff9fc58f640 10 monclient(hunting): authenticate will time out at 7792.271414s 2024-03-25T18:27:57.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.053+0000 7ff9fab2c640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff9f41430c0 0x7ff9f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34034/0 (socket says 172.21.15.42:34034) 2024-03-25T18:27:57.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.053+0000 7ff9fab2c640 1 -- 172.21.15.42:0/1511384594 learned_addr learned my addr 172.21.15.42:0/1511384594 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:57.060 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.059+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418505229 0 0) 0x7ff9f40b28a0 con 0x7ff9f41430c0 2024-03-25T18:27:57.060 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.059+0000 7ff9f9329640 10 monclient(hunting): _negotiate 2024-03-25T18:27:57.060 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.059+0000 7ff9f9329640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:57.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.059+0000 7ff9f9329640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:57.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.059+0000 7ff9f9329640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:27:57.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.059+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e0003520 con 0x7ff9f41430c0 2024-03-25T18:27:57.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.074+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 50598733 0 0) 0x7ff9e0003520 con 0x7ff9f41430c0 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.074+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 >> v1:172.21.15.42:6790/0 conn(0x7ff9f40a98f0 legacy=0x7ff9f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.074+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 >> v1:172.21.15.42:6789/0 conn(0x7ff9f413f710 legacy=0x7ff9f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 10 monclient: _finish_hunting 0 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 1 monclient: found mon.noname-c 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f40b44c0 con 0x7ff9f41430c0 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 10 monclient: _finish_auth 0 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 10 monclient: _check_auth_tickets 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff9ec001a10 con 0x7ff9f41430c0 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9fc58f640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9fc58f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9f9329640 10 monclient: dump: 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.075+0000 7ff9fc58f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:57.088 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9ec001b70 con 0x7ff9f41430c0 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9f9329640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9f9329640 1 -- 172.21.15.42:0/1511384594 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff9ec001f80 con 0x7ff9f41430c0 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9fc58f640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9fc58f640 4 set_mon_vals no callback set 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9f9329640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9f9329640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.087+0000 7ff9f9329640 10 monclient: dump: 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:57.089 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:57.090 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:57.090 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:57.090 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.088+0000 7ff9fc58f640 10 monclient: shutdown 2024-03-25T18:27:57.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.088+0000 7ff9fc58f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:57.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.088+0000 7ff9fc58f640 1 -- 172.21.15.42:0/1511384594 >> v1:172.21.15.120:6789/0 conn(0x7ff9f41430c0 legacy=0x7ff9f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.088+0000 7ff9fc58f640 1 -- 172.21.15.42:0/1511384594 shutdown_connections 2024-03-25T18:27:57.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.088+0000 7ff9fc58f640 1 -- 172.21.15.42:0/1511384594 >> 172.21.15.42:0/1511384594 conn(0x7ff9f40b7ea0 msgr2=0x7ff9f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:57.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.088+0000 7ff9fc58f640 1 -- 172.21.15.42:0/1511384594 shutdown_connections 2024-03-25T18:27:57.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.088+0000 7ff9fc58f640 1 -- 172.21.15.42:0/1511384594 wait complete. 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.089+0000 7ff9fc58f640 10 monclient: build_initial_monmap 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.089+0000 7ff9fc58f640 10 monclient: monmap: 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 librados: starting msgr at 2024-03-25T18:27:57.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 librados: starting objecter 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 Processor -- start 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 -- start start 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 librados: setting wanted keys 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 librados: calling monclient init 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient: init 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:57.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f40a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:27:57.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:27:57.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f41430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:27:57.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f41d7500 con 0x7ff9f41430c0 2024-03-25T18:27:57.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f41d77e0 con 0x7ff9f40a98f0 2024-03-25T18:27:57.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f41d8380 con 0x7ff9f413f710 2024-03-25T18:27:57.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:57.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.090+0000 7ff9fc58f640 10 monclient(hunting): authenticate will time out at 7792.317544s 2024-03-25T18:27:57.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.097+0000 7ff9f9b2a640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ff9f413f710 0x7ff9f41d7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:46370/0 (socket says 172.21.15.42:46370) 2024-03-25T18:27:57.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.098+0000 7ff9f9b2a640 1 -- 172.21.15.42:0/3910014173 learned_addr learned my addr 172.21.15.42:0/3910014173 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:27:57.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.098+0000 7ff9eaffd640 10 client.?.objecter ms_handle_connect 0x7ff9f40a98f0 2024-03-25T18:27:57.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.098+0000 7ff9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:57.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.102+0000 7ff9eaffd640 10 client.?.objecter ms_handle_connect 0x7ff9f41430c0 2024-03-25T18:27:57.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.102+0000 7ff9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.104+0000 7ff9eaffd640 10 client.?.objecter ms_handle_connect 0x7ff9f413f710 2024-03-25T18:27:57.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.104+0000 7ff9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:57.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.106+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207988304 0 0) 0x7ff9f41d77e0 con 0x7ff9f40a98f0 2024-03-25T18:27:57.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.106+0000 7ff9eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:57.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.106+0000 7ff9eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:57.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.106+0000 7ff9eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:57.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.106+0000 7ff9eaffd640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:27:57.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.106+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d8004eb0 con 0x7ff9f40a98f0 2024-03-25T18:27:57.112 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.111+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803782596 0 0) 0x7ff9f41d8380 con 0x7ff9f413f710 2024-03-25T18:27:57.112 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.111+0000 7ff9eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:57.112 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.111+0000 7ff9eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:57.112 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.111+0000 7ff9eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:57.112 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.111+0000 7ff9eaffd640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:27:57.112 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.111+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f41d77e0 con 0x7ff9f413f710 2024-03-25T18:27:57.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.114+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310625922 0 0) 0x7ff9f41d7500 con 0x7ff9f41430c0 2024-03-25T18:27:57.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.114+0000 7ff9eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:57.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.114+0000 7ff9eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:57.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.114+0000 7ff9eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:57.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.114+0000 7ff9eaffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:27:57.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.114+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f41d8380 con 0x7ff9f41430c0 2024-03-25T18:27:57.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3034676873 0 0) 0x7ff9d8004eb0 con 0x7ff9f40a98f0 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f41d7500 con 0x7ff9f40a98f0 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff9f0001930 con 0x7ff9f40a98f0 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 10 monclient(hunting): dump: 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:57.128 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2102115573 0 0) 0x7ff9f41d77e0 con 0x7ff9f413f710 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9d8004eb0 con 0x7ff9f413f710 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff9e4001970 con 0x7ff9f413f710 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.126+0000 7ff9eaffd640 10 monclient(hunting): dump: 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.129 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:57.130 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:57.130 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:57.130 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:57.130 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.138+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2220988695 0 0) 0x7ff9f41d8380 con 0x7ff9f41430c0 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.138+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f41d77e0 con 0x7ff9f41430c0 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.138+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff9ec001e80 con 0x7ff9f41430c0 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.138+0000 7ff9eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.138+0000 7ff9eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.138+0000 7ff9eaffd640 10 monclient(hunting): dump: 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:57.139 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:57.140 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2197466016 0 0) 0x7ff9d8004eb0 con 0x7ff9f413f710 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 >> v1:172.21.15.120:6789/0 conn(0x7ff9f40a98f0 legacy=0x7ff9f4137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 >> v1:172.21.15.42:6789/0 conn(0x7ff9f41430c0 legacy=0x7ff9f4133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 1 monclient: found mon.c 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f40abcc0 con 0x7ff9f413f710 2024-03-25T18:27:57.146 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 10 monclient: _renew_subs 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f41384d0 con 0x7ff9f413f710 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 10 monclient: _renew_subs 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f41350d0 con 0x7ff9f413f710 2024-03-25T18:27:57.147 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.144+0000 7ff9fc58f640 1 librados: init done 2024-03-25T18:27:57.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.146+0000 7ff9fc58f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:57.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.146+0000 7ff9fc58f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:57.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.146+0000 7ff9fc58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:57.148 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.146+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f40627b0 con 0x7ff9f413f710 2024-03-25T18:27:57.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.460+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9e4003610 con 0x7ff9f413f710 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.460+0000 7ff9eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.460+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff9e40039a0 con 0x7ff9f413f710 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.460+0000 7ff9eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.460+0000 7ff9eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.460+0000 7ff9eaffd640 10 monclient: dump: 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:27:57.462 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:27:57.463 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:27:57.463 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:27:57.463 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:27:57.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.460+0000 7ff9f8b28640 4 set_mon_vals no callback set 2024-03-25T18:27:57.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.461+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff9e4003bc0 con 0x7ff9f413f710 2024-03-25T18:27:57.463 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.461+0000 7ff9fa32b640 10 monclient: get_auth_request con 0x7ff9d806e160 auth_method 0 2024-03-25T18:27:57.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.462+0000 7ff9eaffd640 10 client.4604.objecter ms_handle_connect 0x7ff9d806e160 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7ff9e4054740 con 0x7ff9f413f710 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 10 client.4604.objecter ms_dispatch 0x7ff9f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9e4050250 con 0x7ff9f413f710 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:57.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.467+0000 7ff9eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:57.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.693+0000 7ff9fc58f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:57.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.693+0000 7ff9fc58f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:57.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.693+0000 7ff9fc58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:27:57.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.693+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9f41d7990 con 0x7ff9f413f710 2024-03-25T18:27:57.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.735+0000 7ff9eaffd640 1 -- 172.21.15.42:0/3910014173 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7ff9f41d7990 con 0x7ff9f413f710 2024-03-25T18:27:57.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.735+0000 7ff9eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:57.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.735+0000 7ff9eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:57.738 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:27:57.738 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 10 librados: watch_flush enter 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 10 librados: watch_flush exit 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 10 client.4604.objecter successfully canceled tick 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff9d806e160 legacy=0x7ff9d8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 10 monclient: shutdown 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 >> v1:172.21.15.42:6790/0 conn(0x7ff9f413f710 legacy=0x7ff9f41d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 shutdown_connections 2024-03-25T18:27:57.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 >> 172.21.15.42:0/3910014173 conn(0x7ff9f40b7ea0 msgr2=0x7ff9f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:57.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 shutdown_connections 2024-03-25T18:27:57.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 1 -- 172.21.15.42:0/3910014173 wait complete. 2024-03-25T18:27:57.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:27:57.738+0000 7ff9fc58f640 1 librados: shutdown 2024-03-25T18:27:57.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:04.560 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.673+0000 7facda4dd640 10 monclient: get_monmap_and_config 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.673+0000 7facda4dd640 10 monclient: build_initial_monmap 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.673+0000 7facda4dd640 10 monclient: monmap: 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:04.675006+0000 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:04.675006+0000 2024-03-25T18:28:04.675 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:04.676 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:04.676 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:04.676 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:04.676 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:04.676 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:04.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.674+0000 7facda4dd640 1 Processor -- start 2024-03-25T18:28:04.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.674+0000 7facda4dd640 1 -- start start 2024-03-25T18:28:04.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.674+0000 7facda4dd640 10 monclient: init 2024-03-25T18:28:04.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.675+0000 7facda4dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:04.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.675+0000 7facda4dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:04.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.675+0000 7facda4dd640 10 monclient(hunting): picked mon.noname-b con 0x7facd40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:04.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.675+0000 7facda4dd640 10 monclient(hunting): picked mon.noname-a con 0x7facd413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:04.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.675+0000 7facda4dd640 10 monclient(hunting): picked mon.noname-c con 0x7facd41430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:04.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.675+0000 7facda4dd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd4142520 con 0x7facd413f710 2024-03-25T18:28:04.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.676+0000 7facda4dd640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40b28a0 con 0x7facd41430c0 2024-03-25T18:28:04.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.676+0000 7facda4dd640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40b2a30 con 0x7facd40a98f0 2024-03-25T18:28:04.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.676+0000 7facda4dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:04.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.676+0000 7facda4dd640 10 monclient(hunting): authenticate will time out at 7799.902606s 2024-03-25T18:28:04.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.686+0000 7facd37fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7facd413f710 0x7facd41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42710/0 (socket says 172.21.15.42:42710) 2024-03-25T18:28:04.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.686+0000 7facd37fe640 1 -- 172.21.15.42:0/3341746399 learned_addr learned my addr 172.21.15.42:0/3341746399 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:04.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.694+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587207620 0 0) 0x7facd4142520 con 0x7facd413f710 2024-03-25T18:28:04.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.694+0000 7facd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:04.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.694+0000 7facd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:04.695 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.694+0000 7facd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:04.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.694+0000 7facd2ffd640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:28:04.696 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.694+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facbc003520 con 0x7facd413f710 2024-03-25T18:28:04.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.708+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753970379 0 0) 0x7facbc003520 con 0x7facd413f710 2024-03-25T18:28:04.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.708+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 >> v1:172.21.15.42:6790/0 conn(0x7facd40a98f0 legacy=0x7facd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.708+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 >> v1:172.21.15.120:6789/0 conn(0x7facd41430c0 legacy=0x7facd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:04.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40b44c0 con 0x7facd413f710 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7facc8001690 con 0x7facd413f710 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facd2ffd640 10 monclient: dump: 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:04.711 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facda4dd640 5 monclient: authenticate success, global_id 4605 2024-03-25T18:28:04.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.709+0000 7facda4dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:04.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc80017f0 con 0x7facd413f710 2024-03-25T18:28:04.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facd2ffd640 1 -- 172.21.15.42:0/3341746399 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7facc8001c40 con 0x7facd413f710 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facda4dd640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facda4dd640 4 set_mon_vals no callback set 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.718+0000 7facd2ffd640 10 monclient: dump: 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:04.720 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:04.721 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.719+0000 7facda4dd640 10 monclient: shutdown 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.719+0000 7facda4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.719+0000 7facda4dd640 1 -- 172.21.15.42:0/3341746399 >> v1:172.21.15.42:6789/0 conn(0x7facd413f710 legacy=0x7facd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.719+0000 7facda4dd640 1 -- 172.21.15.42:0/3341746399 shutdown_connections 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.719+0000 7facda4dd640 1 -- 172.21.15.42:0/3341746399 >> 172.21.15.42:0/3341746399 conn(0x7facd40b7ea0 msgr2=0x7facd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.719+0000 7facda4dd640 1 -- 172.21.15.42:0/3341746399 shutdown_connections 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.719+0000 7facda4dd640 1 -- 172.21.15.42:0/3341746399 wait complete. 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 10 monclient: build_initial_monmap 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 10 monclient: monmap: 2024-03-25T18:28:04.724 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 1 librados: starting msgr at 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 1 librados: starting objecter 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 1 Processor -- start 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 1 -- start start 2024-03-25T18:28:04.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 1 librados: setting wanted keys 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 1 librados: calling monclient init 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.720+0000 7facda4dd640 10 monclient: init 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 10 monclient(hunting): picked mon.noname-a con 0x7facd40a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 10 monclient(hunting): picked mon.noname-c con 0x7facd413f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 10 monclient(hunting): picked mon.noname-b con 0x7facd41430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40abac0 con 0x7facd40a98f0 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40abda0 con 0x7facd41430c0 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd41380e0 con 0x7facd413f710 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:04.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.721+0000 7facda4dd640 10 monclient(hunting): authenticate will time out at 7799.948120s 2024-03-25T18:28:04.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.730+0000 7facd8a7a640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7facd41430c0 0x7facd41e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:49650/0 (socket says 172.21.15.42:49650) 2024-03-25T18:28:04.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.730+0000 7facd8a7a640 1 -- 172.21.15.42:0/2139378243 learned_addr learned my addr 172.21.15.42:0/2139378243 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:04.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.734+0000 7facd0ff9640 10 client.?.objecter ms_handle_connect 0x7facd41430c0 2024-03-25T18:28:04.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.734+0000 7facd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:04.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.735+0000 7facd0ff9640 10 client.?.objecter ms_handle_connect 0x7facd413f710 2024-03-25T18:28:04.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.735+0000 7facd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:04.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381613688 0 0) 0x7facd41380e0 con 0x7facd413f710 2024-03-25T18:28:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:28:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc0004eb0 con 0x7facd413f710 2024-03-25T18:28:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 10 client.?.objecter ms_handle_connect 0x7facd40a98f0 2024-03-25T18:28:04.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.742+0000 7facd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:04.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.745+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046425671 0 0) 0x7facd40abda0 con 0x7facd41430c0 2024-03-25T18:28:04.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.745+0000 7facd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:04.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.745+0000 7facd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:04.746 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.745+0000 7facd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:04.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.745+0000 7facd0ff9640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:28:04.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.745+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd41380e0 con 0x7facd41430c0 2024-03-25T18:28:04.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.750+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142777948 0 0) 0x7facd40abac0 con 0x7facd40a98f0 2024-03-25T18:28:04.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.750+0000 7facd0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:04.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.750+0000 7facd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:04.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.750+0000 7facd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:04.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.750+0000 7facd0ff9640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:28:04.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.750+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd40abda0 con 0x7facd40a98f0 2024-03-25T18:28:04.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.755+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775635840 0 0) 0x7facc0004eb0 con 0x7facd413f710 2024-03-25T18:28:04.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.756+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facd40abac0 con 0x7facd413f710 2024-03-25T18:28:04.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.756+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7facc8001b00 con 0x7facd413f710 2024-03-25T18:28:04.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.756+0000 7facd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:04.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.756+0000 7facd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:04.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.756+0000 7facd0ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:04.757 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:04.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.756+0000 7facd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.765+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2487869578 0 0) 0x7facd40abda0 con 0x7facd40a98f0 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.765+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc0004eb0 con 0x7facd40a98f0 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.765+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7facc4001830 con 0x7facd40a98f0 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.765+0000 7facd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.765+0000 7facd0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.765+0000 7facd0ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:04.767 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:04.768 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:04.768 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:04.768 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4090488189 0 0) 0x7facc0004eb0 con 0x7facd40a98f0 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 >> v1:172.21.15.42:6790/0 conn(0x7facd413f710 legacy=0x7facd4138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 >> v1:172.21.15.120:6789/0 conn(0x7facd41430c0 legacy=0x7facd41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 1 monclient: found mon.a 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4137d90 con 0x7facd40a98f0 2024-03-25T18:28:04.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 5 monclient: authenticate success, global_id 4608 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 10 monclient: _renew_subs 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd4139240 con 0x7facd40a98f0 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 10 monclient: _renew_subs 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facd4136830 con 0x7facd40a98f0 2024-03-25T18:28:04.774 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.771+0000 7facda4dd640 1 librados: init done 2024-03-25T18:28:04.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.772+0000 7facda4dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:04.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.772+0000 7facda4dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:04.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.772+0000 7facda4dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:04.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:04.772+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd40627b0 con 0x7facd40a98f0 2024-03-25T18:28:05.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.097+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc4001d40 con 0x7facd40a98f0 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.097+0000 7facd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.097+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7facc4003790 con 0x7facd40a98f0 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.097+0000 7facd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.097+0000 7facd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.097+0000 7facd0ff9640 10 monclient: dump: 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:05.099 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:05.100 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:05.100 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:05.100 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:05.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.097+0000 7facd27fc640 4 set_mon_vals no callback set 2024-03-25T18:28:05.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.098+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7facc40039b0 con 0x7facd40a98f0 2024-03-25T18:28:05.100 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.098+0000 7facd37fe640 10 monclient: get_auth_request con 0x7facc006df70 auth_method 0 2024-03-25T18:28:05.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.099+0000 7facd0ff9640 10 client.4608.objecter ms_handle_connect 0x7facc006df70 2024-03-25T18:28:05.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7facc40535b0 con 0x7facd40a98f0 2024-03-25T18:28:05.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 10 client.4608.objecter ms_dispatch 0x7facd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:05.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 3 client.4608.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:05.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 3 client.4608.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:05.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-25T18:28:05.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facc404eef0 con 0x7facd40a98f0 2024-03-25T18:28:05.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:05.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.104+0000 7facd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:05.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.368+0000 7facda4dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:05.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.368+0000 7facda4dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:05.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.368+0000 7facda4dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:05.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.368+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facd413f070 con 0x7facd40a98f0 2024-03-25T18:28:05.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.394+0000 7facd0ff9640 1 -- 172.21.15.42:0/2139378243 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7facc4050ff0 con 0x7facd40a98f0 2024-03-25T18:28:05.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.394+0000 7facd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:05.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.394+0000 7facd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:05.396 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:05.396 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 10 librados: watch_flush enter 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 10 librados: watch_flush exit 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 10 client.4608.objecter successfully canceled tick 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 >> v1:172.21.15.42:6800/3081750888 conn(0x7facc006df70 legacy=0x7facc0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 10 monclient: shutdown 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 >> v1:172.21.15.42:6789/0 conn(0x7facd40a98f0 legacy=0x7facd40ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 shutdown_connections 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.396+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 >> 172.21.15.42:0/2139378243 conn(0x7facd40b7ea0 msgr2=0x7facd4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.397+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 shutdown_connections 2024-03-25T18:28:05.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.397+0000 7facda4dd640 1 -- 172.21.15.42:0/2139378243 wait complete. 2024-03-25T18:28:05.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:05.397+0000 7facda4dd640 1 librados: shutdown 2024-03-25T18:28:05.411 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:12.217 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:12.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.326+0000 7f751607b640 10 monclient: get_monmap_and_config 2024-03-25T18:28:12.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.326+0000 7f751607b640 10 monclient: build_initial_monmap 2024-03-25T18:28:12.327 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.326+0000 7f751607b640 10 monclient: monmap: 2024-03-25T18:28:12.327 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:12.327233+0000 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:12.327233+0000 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:12.328 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:12.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.326+0000 7f751607b640 1 Processor -- start 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.326+0000 7f751607b640 1 -- start start 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.326+0000 7f751607b640 10 monclient: init 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 10 monclient(hunting): picked mon.noname-b con 0x7f75100a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 10 monclient(hunting): picked mon.noname-a con 0x7f751013f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 10 monclient(hunting): picked mon.noname-c con 0x7f75101430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7510142520 con 0x7f751013f710 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100b28a0 con 0x7f75101430c0 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100b2a30 con 0x7f75100a98f0 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:12.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.327+0000 7f751607b640 10 monclient(hunting): authenticate will time out at 7807.554189s 2024-03-25T18:28:12.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.331+0000 7f750f7fe640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f75100a98f0 0x7f75100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:50980/0 (socket says 172.21.15.42:50980) 2024-03-25T18:28:12.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.331+0000 7f750f7fe640 1 -- 172.21.15.42:0/1812788073 learned_addr learned my addr 172.21.15.42:0/1812788073 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:12.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.340+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457179396 0 0) 0x7f75100b2a30 con 0x7f75100a98f0 2024-03-25T18:28:12.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.340+0000 7f750e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:12.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.340+0000 7f750e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:12.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.340+0000 7f750e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:12.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.340+0000 7f750e7fc640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:28:12.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.340+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8003520 con 0x7f75100a98f0 2024-03-25T18:28:12.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.346+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391795533 0 0) 0x7f75100b28a0 con 0x7f75101430c0 2024-03-25T18:28:12.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.346+0000 7f750e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:12.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.346+0000 7f750e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:12.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.346+0000 7f750e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:12.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.346+0000 7f750e7fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:28:12.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.346+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75100b2a30 con 0x7f75101430c0 2024-03-25T18:28:12.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.365+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021575715 0 0) 0x7f7510142520 con 0x7f751013f710 2024-03-25T18:28:12.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.365+0000 7f750e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:12.366 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.365+0000 7f750e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:12.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.365+0000 7f750e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:12.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.365+0000 7f750e7fc640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:28:12.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.365+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75100b28a0 con 0x7f751013f710 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924824282 0 0) 0x7f74f8003520 con 0x7f75100a98f0 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 >> v1:172.21.15.120:6789/0 conn(0x7f75101430c0 legacy=0x7f751014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 >> v1:172.21.15.42:6789/0 conn(0x7f751013f710 legacy=0x7f75101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75100b44c0 con 0x7f75100a98f0 2024-03-25T18:28:12.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7500001690 con 0x7f75100a98f0 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f750e7fc640 10 monclient: dump: 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:12.375 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:12.376 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:12.376 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:12.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f751607b640 5 monclient: authenticate success, global_id 4613 2024-03-25T18:28:12.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.372+0000 7f751607b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:12.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75000017f0 con 0x7f75100a98f0 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f750e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f750e7fc640 1 -- 172.21.15.42:0/1812788073 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7500001c00 con 0x7f75100a98f0 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f751607b640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f751607b640 4 set_mon_vals no callback set 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f750e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f750e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.393+0000 7f750e7fc640 10 monclient: dump: 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:12.395 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:12.396 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:12.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.394+0000 7f751607b640 10 monclient: shutdown 2024-03-25T18:28:12.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.394+0000 7f751607b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:12.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.394+0000 7f751607b640 1 -- 172.21.15.42:0/1812788073 >> v1:172.21.15.42:6790/0 conn(0x7f75100a98f0 legacy=0x7f75100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:12.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.394+0000 7f751607b640 1 -- 172.21.15.42:0/1812788073 shutdown_connections 2024-03-25T18:28:12.398 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.394+0000 7f751607b640 1 -- 172.21.15.42:0/1812788073 >> 172.21.15.42:0/1812788073 conn(0x7f75100b7ea0 msgr2=0x7f75100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.394+0000 7f751607b640 1 -- 172.21.15.42:0/1812788073 shutdown_connections 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.394+0000 7f751607b640 1 -- 172.21.15.42:0/1812788073 wait complete. 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 10 monclient: build_initial_monmap 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 10 monclient: monmap: 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:12.399 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 1 librados: starting msgr at 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 1 librados: starting objecter 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 1 Processor -- start 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 1 -- start start 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 1 librados: setting wanted keys 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 1 librados: calling monclient init 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.395+0000 7f751607b640 10 monclient: init 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 10 monclient(hunting): picked mon.noname-c con 0x7f75100a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 10 monclient(hunting): picked mon.noname-a con 0x7f751013f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 10 monclient(hunting): picked mon.noname-b con 0x7f75101430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100abd30 con 0x7f751013f710 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100abfa0 con 0x7f75101430c0 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75101d77a0 con 0x7f75100a98f0 2024-03-25T18:28:12.400 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:12.401 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.396+0000 7f751607b640 10 monclient(hunting): authenticate will time out at 7807.623314s 2024-03-25T18:28:12.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.403+0000 7f750ffff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f75101430c0 0x7f75101385d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:36504/0 (socket says 172.21.15.42:36504) 2024-03-25T18:28:12.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.403+0000 7f750ffff640 1 -- 172.21.15.42:0/2989305630 learned_addr learned my addr 172.21.15.42:0/2989305630 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:12.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.407+0000 7f74f7fff640 10 client.?.objecter ms_handle_connect 0x7f75101430c0 2024-03-25T18:28:12.408 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.407+0000 7f74f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:12.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.409+0000 7f74f7fff640 10 client.?.objecter ms_handle_connect 0x7f751013f710 2024-03-25T18:28:12.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.409+0000 7f74f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:12.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.415+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523717432 0 0) 0x7f75100abd30 con 0x7f751013f710 2024-03-25T18:28:12.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.415+0000 7f74f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:12.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.415+0000 7f74f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:12.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.415+0000 7f74f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:12.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.415+0000 7f74f7fff640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:28:12.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.415+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74fc004eb0 con 0x7f751013f710 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 10 client.?.objecter ms_handle_connect 0x7f75100a98f0 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578937386 0 0) 0x7f75100abfa0 con 0x7f75101430c0 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:28:12.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.416+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75100abd30 con 0x7f75101430c0 2024-03-25T18:28:12.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.424+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020418492 0 0) 0x7f75101d77a0 con 0x7f75100a98f0 2024-03-25T18:28:12.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.424+0000 7f74f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:12.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.424+0000 7f74f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:12.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.424+0000 7f74f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:12.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.424+0000 7f74f7fff640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:28:12.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.425+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75100abfa0 con 0x7f75100a98f0 2024-03-25T18:28:12.431 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.430+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358828010 0 0) 0x7f75100abd30 con 0x7f75101430c0 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.430+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75101d77a0 con 0x7f75101430c0 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.430+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7508001b70 con 0x7f75101430c0 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.430+0000 7f74f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.430+0000 7f74f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.430+0000 7f74f7fff640 10 monclient(hunting): dump: 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:12.432 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:12.433 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:12.433 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:12.433 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:12.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.430+0000 7f74f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.438+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1507625786 0 0) 0x7f75100abfa0 con 0x7f75100a98f0 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.438+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75100abd30 con 0x7f75100a98f0 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.438+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7500001b00 con 0x7f75100a98f0 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.438+0000 7f74f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.438+0000 7f74f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.438+0000 7f74f7fff640 10 monclient(hunting): dump: 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:12.439 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:12.440 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:12.442 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1685822895 0 0) 0x7f75101d77a0 con 0x7f75101430c0 2024-03-25T18:28:12.442 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 >> v1:172.21.15.42:6790/0 conn(0x7f75100a98f0 legacy=0x7f75101d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:12.442 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 >> v1:172.21.15.42:6789/0 conn(0x7f751013f710 legacy=0x7f75101d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:12.442 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:12.442 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 1 monclient: found mon.b 2024-03-25T18:28:12.442 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:12.442 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75101d83d0 con 0x7f75101430c0 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f74f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 10 monclient: _renew_subs 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75101d41e0 con 0x7f75101430c0 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 10 monclient: _renew_subs 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75101d5110 con 0x7f75101430c0 2024-03-25T18:28:12.443 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.440+0000 7f751607b640 1 librados: init done 2024-03-25T18:28:12.444 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.441+0000 7f751607b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:12.444 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.441+0000 7f751607b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:12.444 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.441+0000 7f751607b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:12.444 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.441+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75101d7930 con 0x7f75101430c0 2024-03-25T18:28:12.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.729+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7508002080 con 0x7f75101430c0 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.729+0000 7f74f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.729+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f7508003b30 con 0x7f75101430c0 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.729+0000 7f74f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.729+0000 7f74f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.729+0000 7f74f7fff640 10 monclient: dump: 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:12.731 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:12.732 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:12.732 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:12.732 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:12.732 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:12.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.729+0000 7f750dffb640 4 set_mon_vals no callback set 2024-03-25T18:28:12.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.730+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f7508003d50 con 0x7f75101430c0 2024-03-25T18:28:12.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.730+0000 7f750f7fe640 10 monclient: get_auth_request con 0x7f74fc06dee0 auth_method 0 2024-03-25T18:28:12.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.730+0000 7f74f7fff640 10 client.4621.objecter ms_handle_connect 0x7f74fc06dee0 2024-03-25T18:28:12.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f7508054a70 con 0x7f75101430c0 2024-03-25T18:28:12.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 10 client.4621.objecter ms_dispatch 0x7f7510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:12.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:12.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:12.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T18:28:12.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75080503b0 con 0x7f75101430c0 2024-03-25T18:28:12.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:12.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.735+0000 7f74f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:12.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.993+0000 7f751607b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:12.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.993+0000 7f751607b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:12.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.993+0000 7f751607b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:12.994 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:12.993+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7510136a20 con 0x7f75101430c0 2024-03-25T18:28:13.055 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:56.224 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.063 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:56.232 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.102 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:08:56.271 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.118+0000 7f74f7fff640 1 -- 172.21.15.42:0/2989305630 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f75080524b0 con 0x7f75101430c0 2024-03-25T18:28:13.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.118+0000 7f74f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:13.119 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.118+0000 7f74f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:13.120 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:13.120 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:13.120 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:56.240 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 10 librados: watch_flush enter 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 10 librados: watch_flush exit 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 10 client.4621.objecter successfully canceled tick 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 >> v1:172.21.15.42:6800/3081750888 conn(0x7f74fc06dee0 legacy=0x7f74fc0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 10 monclient: shutdown 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 >> v1:172.21.15.120:6789/0 conn(0x7f75101430c0 legacy=0x7f75101385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:13.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 shutdown_connections 2024-03-25T18:28:13.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 >> 172.21.15.42:0/2989305630 conn(0x7f75100b7ea0 msgr2=0x7f751013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:13.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 shutdown_connections 2024-03-25T18:28:13.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 1 -- 172.21.15.42:0/2989305630 wait complete. 2024-03-25T18:28:13.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:13.120+0000 7f751607b640 1 librados: shutdown 2024-03-25T18:28:13.127 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:56.297 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.132 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:56.252 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:13.138 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:56.308 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.154 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:08:56.274 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:13.157 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:08:56.327 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.940 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.055+0000 7fa366086640 10 monclient: get_monmap_and_config 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.055+0000 7fa366086640 10 monclient: build_initial_monmap 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.055+0000 7fa366086640 10 monclient: monmap: 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:20.056779+0000 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:20.056779+0000 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:20.057 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:20.058 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:20.058 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:20.058 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:20.058 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.056+0000 7fa366086640 1 Processor -- start 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.056+0000 7fa366086640 1 -- start start 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.056+0000 7fa366086640 10 monclient: init 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 10 monclient(hunting): picked mon.noname-a con 0x7fa3600a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 10 monclient(hunting): picked mon.noname-c con 0x7fa36013f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 10 monclient(hunting): picked mon.noname-b con 0x7fa3601430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa360142520 con 0x7fa3600a98f0 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3600b28a0 con 0x7fa36013f710 2024-03-25T18:28:20.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3600b2a30 con 0x7fa3601430c0 2024-03-25T18:28:20.060 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:20.060 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.057+0000 7fa366086640 10 monclient(hunting): authenticate will time out at 7815.284453s 2024-03-25T18:28:20.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.074+0000 7fa35effd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fa36013f710 0x7fa3601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:52170/0 (socket says 172.21.15.42:52170) 2024-03-25T18:28:20.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.074+0000 7fa35effd640 1 -- 172.21.15.42:0/3710351916 learned_addr learned my addr 172.21.15.42:0/3710351916 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:20.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.078+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644297181 0 0) 0x7fa3600b28a0 con 0x7fa36013f710 2024-03-25T18:28:20.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.078+0000 7fa35e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:20.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.078+0000 7fa35e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:20.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.078+0000 7fa35e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:20.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.078+0000 7fa35e7fc640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:28:20.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.078+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa348003520 con 0x7fa36013f710 2024-03-25T18:28:20.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.087+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1479706466 0 0) 0x7fa348003520 con 0x7fa36013f710 2024-03-25T18:28:20.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 >> v1:172.21.15.42:6790/0 conn(0x7fa3601430c0 legacy=0x7fa36014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:20.089 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 >> v1:172.21.15.42:6789/0 conn(0x7fa3600a98f0 legacy=0x7fa3600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600b44c0 con 0x7fa36013f710 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa354001690 con 0x7fa36013f710 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:20.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa35e7fc640 10 monclient: dump: 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa366086640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:28:20.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.088+0000 7fa366086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3540017f0 con 0x7fa36013f710 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa35e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa35e7fc640 1 -- 172.21.15.42:0/3710351916 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa354001c40 con 0x7fa36013f710 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa366086640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa366086640 4 set_mon_vals no callback set 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa35e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa35e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.101+0000 7fa35e7fc640 10 monclient: dump: 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.103 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.104 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:20.104 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:20.104 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:20.104 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:20.104 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:20.104 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:20.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.102+0000 7fa366086640 10 monclient: shutdown 2024-03-25T18:28:20.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.102+0000 7fa366086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:20.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.102+0000 7fa366086640 1 -- 172.21.15.42:0/3710351916 >> v1:172.21.15.120:6789/0 conn(0x7fa36013f710 legacy=0x7fa3601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:20.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.102+0000 7fa366086640 1 -- 172.21.15.42:0/3710351916 shutdown_connections 2024-03-25T18:28:20.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.102+0000 7fa366086640 1 -- 172.21.15.42:0/3710351916 >> 172.21.15.42:0/3710351916 conn(0x7fa3600b7ea0 msgr2=0x7fa3600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:20.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.102+0000 7fa366086640 1 -- 172.21.15.42:0/3710351916 shutdown_connections 2024-03-25T18:28:20.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.103+0000 7fa366086640 1 -- 172.21.15.42:0/3710351916 wait complete. 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.103+0000 7fa366086640 10 monclient: build_initial_monmap 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.103+0000 7fa366086640 10 monclient: monmap: 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:20.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.104+0000 7fa366086640 1 librados: starting msgr at 2024-03-25T18:28:20.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.104+0000 7fa366086640 1 librados: starting objecter 2024-03-25T18:28:20.107 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.104+0000 7fa366086640 1 Processor -- start 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.104+0000 7fa366086640 1 -- start start 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.104+0000 7fa366086640 1 librados: setting wanted keys 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.104+0000 7fa366086640 1 librados: calling monclient init 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.104+0000 7fa366086640 10 monclient: init 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 10 monclient(hunting): picked mon.noname-a con 0x7fa3600a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 10 monclient(hunting): picked mon.noname-b con 0x7fa36013f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 10 monclient(hunting): picked mon.noname-c con 0x7fa3601430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:20.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3601d51f0 con 0x7fa3600a98f0 2024-03-25T18:28:20.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa360135470 con 0x7fa36013f710 2024-03-25T18:28:20.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3601360a0 con 0x7fa3601430c0 2024-03-25T18:28:20.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:20.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.105+0000 7fa366086640 10 monclient(hunting): authenticate will time out at 7815.331981s 2024-03-25T18:28:20.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.114+0000 7fa35f7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa3600a98f0 0x7fa360135990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57884/0 (socket says 172.21.15.42:57884) 2024-03-25T18:28:20.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.114+0000 7fa35f7fe640 1 -- 172.21.15.42:0/1314386762 learned_addr learned my addr 172.21.15.42:0/1314386762 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:20.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.114+0000 7fa347fff640 10 client.?.objecter ms_handle_connect 0x7fa3600a98f0 2024-03-25T18:28:20.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.114+0000 7fa347fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:20.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.120+0000 7fa347fff640 10 client.?.objecter ms_handle_connect 0x7fa36013f710 2024-03-25T18:28:20.122 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.120+0000 7fa347fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:20.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.122+0000 7fa347fff640 10 client.?.objecter ms_handle_connect 0x7fa3601430c0 2024-03-25T18:28:20.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.122+0000 7fa347fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:20.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.127+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281704455 0 0) 0x7fa3601d51f0 con 0x7fa3600a98f0 2024-03-25T18:28:20.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.127+0000 7fa347fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:20.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.127+0000 7fa347fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:20.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.127+0000 7fa347fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:20.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.127+0000 7fa347fff640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:28:20.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.127+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34c004eb0 con 0x7fa3600a98f0 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.135+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665745876 0 0) 0x7fa34c004eb0 con 0x7fa3600a98f0 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.135+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3601d51f0 con 0x7fa3600a98f0 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.135+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa3500019d0 con 0x7fa3600a98f0 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.135+0000 7fa347fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.135+0000 7fa347fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.135+0000 7fa347fff640 10 monclient(hunting): dump: 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.137 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:20.138 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:20.138 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:20.138 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:20.138 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:20.138 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:20.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.135+0000 7fa347fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:20.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.141+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275410173 0 0) 0x7fa3601360a0 con 0x7fa3601430c0 2024-03-25T18:28:20.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.141+0000 7fa347fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:20.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.141+0000 7fa347fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:20.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.141+0000 7fa347fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:20.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.141+0000 7fa347fff640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:28:20.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.141+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34c004eb0 con 0x7fa3601430c0 2024-03-25T18:28:20.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.142+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384928514 0 0) 0x7fa360135470 con 0x7fa36013f710 2024-03-25T18:28:20.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.142+0000 7fa347fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:20.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.142+0000 7fa347fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:20.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.142+0000 7fa347fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:20.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.142+0000 7fa347fff640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:28:20.144 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.142+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3601360a0 con 0x7fa36013f710 2024-03-25T18:28:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4186453323 0 0) 0x7fa3601d51f0 con 0x7fa3600a98f0 2024-03-25T18:28:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 >> v1:172.21.15.42:6790/0 conn(0x7fa3601430c0 legacy=0x7fa3601db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 >> v1:172.21.15.120:6789/0 conn(0x7fa36013f710 legacy=0x7fa3601dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 1 monclient: found mon.a 2024-03-25T18:28:20.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360135100 con 0x7fa3600a98f0 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa366086640 5 monclient: authenticate success, global_id 4620 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa366086640 10 monclient: _renew_subs 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa366086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.149+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3601e2e40 con 0x7fa3600a98f0 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.150+0000 7fa366086640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:20.152 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.150+0000 7fa366086640 10 monclient: _renew_subs 2024-03-25T18:28:20.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.150+0000 7fa366086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:20.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.150+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3601e31b0 con 0x7fa3600a98f0 2024-03-25T18:28:20.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.150+0000 7fa366086640 1 librados: init done 2024-03-25T18:28:20.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.151+0000 7fa366086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:20.153 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.151+0000 7fa366086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:20.154 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.151+0000 7fa366086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:20.154 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.151+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3600627b0 con 0x7fa3600a98f0 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.457+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa350001c50 con 0x7fa3600a98f0 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.457+0000 7fa347fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.457+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fa350003a60 con 0x7fa3600a98f0 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.457+0000 7fa347fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.457+0000 7fa347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.457+0000 7fa347fff640 10 monclient: dump: 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:20.459 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:20.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.457+0000 7fa35dffb640 4 set_mon_vals no callback set 2024-03-25T18:28:20.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.458+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fa350003c80 con 0x7fa3600a98f0 2024-03-25T18:28:20.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.459+0000 7fa35effd640 10 monclient: get_auth_request con 0x7fa34c06de10 auth_method 0 2024-03-25T18:28:20.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.459+0000 7fa347fff640 10 client.4620.objecter ms_handle_connect 0x7fa34c06de10 2024-03-25T18:28:20.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fa350054830 con 0x7fa3600a98f0 2024-03-25T18:28:20.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 10 client.4620.objecter ms_dispatch 0x7fa360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:20.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 3 client.4620.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:20.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 3 client.4620.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:20.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-25T18:28:20.481 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa350050340 con 0x7fa3600a98f0 2024-03-25T18:28:20.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:20.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.479+0000 7fa347fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:20.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.705+0000 7fa366086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:20.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.705+0000 7fa366086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:20.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.705+0000 7fa366086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:20.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.705+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa360135600 con 0x7fa3600a98f0 2024-03-25T18:28:20.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.726+0000 7fa347fff640 1 -- 172.21.15.42:0/1314386762 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fa350052270 con 0x7fa3600a98f0 2024-03-25T18:28:20.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.726+0000 7fa347fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:20.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.726+0000 7fa347fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:20.727 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:20.728 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:20.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.727+0000 7fa366086640 10 librados: watch_flush enter 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 10 librados: watch_flush exit 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 10 client.4620.objecter successfully canceled tick 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 >> v1:172.21.15.42:6800/3081750888 conn(0x7fa34c06de10 legacy=0x7fa34c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 10 monclient: shutdown 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 >> v1:172.21.15.42:6789/0 conn(0x7fa3600a98f0 legacy=0x7fa360135990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 shutdown_connections 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 >> 172.21.15.42:0/1314386762 conn(0x7fa3600b7ea0 msgr2=0x7fa360074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 shutdown_connections 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 1 -- 172.21.15.42:0/1314386762 wait complete. 2024-03-25T18:28:20.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:20.728+0000 7fa366086640 1 librados: shutdown 2024-03-25T18:28:20.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:27.548 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:27.663 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.662+0000 7fc10b21e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.662+0000 7fc10b21e640 10 monclient: build_initial_monmap 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.662+0000 7fc10b21e640 10 monclient: monmap: 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:27.663571+0000 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:27.663571+0000 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:27.664 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 1 Processor -- start 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 1 -- start start 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 10 monclient: init 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 10 monclient(hunting): picked mon.noname-b con 0x7fc1040b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 10 monclient(hunting): picked mon.noname-c con 0x7fc1040b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 10 monclient(hunting): picked mon.noname-a con 0x7fc10414be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.663+0000 7fc10b21e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1040af940 con 0x7fc10414be90 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.664+0000 7fc10b21e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1040b0a70 con 0x7fc1040b1730 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.664+0000 7fc10b21e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1040b0d50 con 0x7fc1040b4d60 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.664+0000 7fc10b21e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:27.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.664+0000 7fc10b21e640 10 monclient(hunting): authenticate will time out at 7822.890586s 2024-03-25T18:28:27.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.676+0000 7fc1097bb640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc10414be90 0x7fc10414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58170/0 (socket says 172.21.15.42:58170) 2024-03-25T18:28:27.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.676+0000 7fc1097bb640 1 -- 172.21.15.42:0/2504025426 learned_addr learned my addr 172.21.15.42:0/2504025426 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:27.686 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.685+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659034675 0 0) 0x7fc1040af940 con 0x7fc10414be90 2024-03-25T18:28:27.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.685+0000 7fc1037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:27.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.685+0000 7fc1037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:27.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.685+0000 7fc1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:27.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.685+0000 7fc1037fe640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:28:27.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.685+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f4003520 con 0x7fc10414be90 2024-03-25T18:28:27.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 465389918 0 0) 0x7fc0f4003520 con 0x7fc10414be90 2024-03-25T18:28:27.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 >> v1:172.21.15.42:6790/0 conn(0x7fc1040b4d60 legacy=0x7fc10413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:27.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 >> v1:172.21.15.120:6789/0 conn(0x7fc1040b1730 legacy=0x7fc1040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:27.697 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 1 monclient: found mon.noname-a 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1040b1030 con 0x7fc10414be90 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc0fc001a10 con 0x7fc10414be90 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc10b21e640 5 monclient: authenticate success, global_id 4623 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc10b21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:27.698 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc1037fe640 10 monclient: dump: 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:27.699 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.695+0000 7fc10b21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:27.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.702+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0fc001b70 con 0x7fc10414be90 2024-03-25T18:28:27.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.702+0000 7fc1037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:27.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.702+0000 7fc10b21e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:27.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.702+0000 7fc10b21e640 4 set_mon_vals no callback set 2024-03-25T18:28:27.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc10b21e640 10 monclient: shutdown 2024-03-25T18:28:27.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc1037fe640 1 -- 172.21.15.42:0/2504025426 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc0fc001f80 con 0x7fc10414be90 2024-03-25T18:28:27.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc1037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:27.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc1037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:27.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc1037fe640 10 monclient: dump: 2024-03-25T18:28:27.705 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:27.705 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc10b21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc10b21e640 1 -- 172.21.15.42:0/2504025426 >> v1:172.21.15.42:6789/0 conn(0x7fc10414be90 legacy=0x7fc10414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc10b21e640 1 -- 172.21.15.42:0/2504025426 shutdown_connections 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.703+0000 7fc10b21e640 1 -- 172.21.15.42:0/2504025426 >> 172.21.15.42:0/2504025426 conn(0x7fc1040617a0 msgr2=0x7fc104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:27.706 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.704+0000 7fc10b21e640 1 -- 172.21.15.42:0/2504025426 shutdown_connections 2024-03-25T18:28:27.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.704+0000 7fc10b21e640 1 -- 172.21.15.42:0/2504025426 wait complete. 2024-03-25T18:28:27.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.704+0000 7fc10b21e640 10 monclient: build_initial_monmap 2024-03-25T18:28:27.708 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.704+0000 7fc10b21e640 10 monclient: monmap: 2024-03-25T18:28:27.708 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:27.708 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:27.708 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:27.708 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:27.708 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 1 librados: starting msgr at 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 1 librados: starting objecter 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 1 Processor -- start 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 1 -- start start 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 1 librados: setting wanted keys 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 1 librados: calling monclient init 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 10 monclient: init 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:27.709 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 10 monclient(hunting): picked mon.noname-b con 0x7fc1040b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.705+0000 7fc10b21e640 10 monclient(hunting): picked mon.noname-a con 0x7fc1040b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.706+0000 7fc10b21e640 10 monclient(hunting): picked mon.noname-c con 0x7fc10414be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.706+0000 7fc10b21e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1041d7840 con 0x7fc1040b4d60 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.706+0000 7fc10b21e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc104138890 con 0x7fc1040b1730 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.706+0000 7fc10b21e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1041d8470 con 0x7fc10414be90 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.706+0000 7fc10b21e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:27.710 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.706+0000 7fc10b21e640 10 monclient(hunting): authenticate will time out at 7822.932653s 2024-03-25T18:28:27.717 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.716+0000 7fc1097bb640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fc10414be90 0x7fc104142a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:33910/0 (socket says 172.21.15.42:33910) 2024-03-25T18:28:27.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.716+0000 7fc1097bb640 1 -- 172.21.15.42:0/2617376322 learned_addr learned my addr 172.21.15.42:0/2617376322 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:27.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.717+0000 7fc1017fa640 10 client.?.objecter ms_handle_connect 0x7fc10414be90 2024-03-25T18:28:27.719 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.717+0000 7fc1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:27.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.722+0000 7fc1017fa640 10 client.?.objecter ms_handle_connect 0x7fc1040b4d60 2024-03-25T18:28:27.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.722+0000 7fc1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:27.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.727+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076547456 0 0) 0x7fc1041d7840 con 0x7fc1040b4d60 2024-03-25T18:28:27.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.727+0000 7fc1017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:27.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.727+0000 7fc1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:27.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.727+0000 7fc1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:27.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.727+0000 7fc1017fa640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:28:27.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.727+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e8004eb0 con 0x7fc1040b4d60 2024-03-25T18:28:27.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.728+0000 7fc1017fa640 10 client.?.objecter ms_handle_connect 0x7fc1040b1730 2024-03-25T18:28:27.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.728+0000 7fc1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:27.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.732+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460309118 0 0) 0x7fc1041d8470 con 0x7fc10414be90 2024-03-25T18:28:27.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.732+0000 7fc1017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:27.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.732+0000 7fc1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:27.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.732+0000 7fc1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:27.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.732+0000 7fc1017fa640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:28:27.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.732+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1041d7840 con 0x7fc10414be90 2024-03-25T18:28:27.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.739+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524133969 0 0) 0x7fc0e8004eb0 con 0x7fc1040b4d60 2024-03-25T18:28:27.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.739+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1041d8470 con 0x7fc1040b4d60 2024-03-25T18:28:27.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.739+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc0f00018c0 con 0x7fc1040b4d60 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.739+0000 7fc1017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.739+0000 7fc1017fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.739+0000 7fc1017fa640 10 monclient(hunting): dump: 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:27.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.739+0000 7fc1017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:27.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.741+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851705696 0 0) 0x7fc104138890 con 0x7fc1040b1730 2024-03-25T18:28:27.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.741+0000 7fc1017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:27.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.741+0000 7fc1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:27.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.741+0000 7fc1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:27.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.741+0000 7fc1017fa640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:28:27.742 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.741+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e8004eb0 con 0x7fc1040b1730 2024-03-25T18:28:27.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 870799432 0 0) 0x7fc1041d8470 con 0x7fc1040b4d60 2024-03-25T18:28:27.747 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 >> v1:172.21.15.42:6790/0 conn(0x7fc10414be90 legacy=0x7fc104142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 >> v1:172.21.15.120:6789/0 conn(0x7fc1040b1730 legacy=0x7fc104138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 1 monclient: found mon.a 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1041d8600 con 0x7fc1040b4d60 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc1017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 5 monclient: authenticate success, global_id 4626 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 10 monclient: _renew_subs 2024-03-25T18:28:27.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc104142330 con 0x7fc1040b4d60 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 10 monclient: _renew_subs 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1041432f0 con 0x7fc1040b4d60 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.746+0000 7fc10b21e640 1 librados: init done 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.747+0000 7fc10b21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:27.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.747+0000 7fc10b21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:27.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.747+0000 7fc10b21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:27.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.747+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1041d79f0 con 0x7fc1040b4d60 2024-03-25T18:28:27.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.769+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0f00035f0 con 0x7fc1040b4d60 2024-03-25T18:28:27.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.769+0000 7fc1017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:27.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.769+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc0f0003860 con 0x7fc1040b4d60 2024-03-25T18:28:27.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.769+0000 7fc1017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.769+0000 7fc1017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.769+0000 7fc1017fa640 10 monclient: dump: 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:27.771 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:27.772 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:27.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.769+0000 7fc102ffd640 4 set_mon_vals no callback set 2024-03-25T18:28:27.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.770+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fc0f0003a80 con 0x7fc1040b4d60 2024-03-25T18:28:27.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.770+0000 7fc108fba640 10 monclient: get_auth_request con 0x7fc0e806de00 auth_method 0 2024-03-25T18:28:27.773 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:27.771+0000 7fc1017fa640 10 client.4626.objecter ms_handle_connect 0x7fc0e806de00 2024-03-25T18:28:28.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fc0f00533a0 con 0x7fc1040b4d60 2024-03-25T18:28:28.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 10 client.4626.objecter ms_dispatch 0x7fc104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:28.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:28.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:28.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T18:28:28.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0f004eeb0 con 0x7fc1040b4d60 2024-03-25T18:28:28.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:28.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.077+0000 7fc1017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:28.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.303+0000 7fc10b21e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:28.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.303+0000 7fc10b21e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:28.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.303+0000 7fc10b21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:28:28.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.303+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc104141ff0 con 0x7fc1040b4d60 2024-03-25T18:28:28.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.326+0000 7fc1017fa640 1 -- 172.21.15.42:0/2617376322 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fc0f0050de0 con 0x7fc1040b4d60 2024-03-25T18:28:28.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.326+0000 7fc1017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:28.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.326+0000 7fc1017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:28.328 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:28.328 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:28.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.328+0000 7fc10b21e640 10 librados: watch_flush enter 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 10 librados: watch_flush exit 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 10 client.4626.objecter successfully canceled tick 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 >> v1:172.21.15.42:6800/3081750888 conn(0x7fc0e806de00 legacy=0x7fc0e80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 10 monclient: shutdown 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 >> v1:172.21.15.42:6789/0 conn(0x7fc1040b4d60 legacy=0x7fc1041d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 shutdown_connections 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 >> 172.21.15.42:0/2617376322 conn(0x7fc1040617a0 msgr2=0x7fc10413c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 shutdown_connections 2024-03-25T18:28:28.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 1 -- 172.21.15.42:0/2617376322 wait complete. 2024-03-25T18:28:28.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:28.329+0000 7fc10b21e640 1 librados: shutdown 2024-03-25T18:28:28.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:33.202 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:16.371 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.208 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:16.377 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.258 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:16.426 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.260 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:16.380 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.269 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:16.389 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.270 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:16.439 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.275 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:16.444 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.287 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:16.407 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:33.306 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:16.475 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:35.150 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:35.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.267+0000 7fc908486640 10 monclient: get_monmap_and_config 2024-03-25T18:28:35.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.267+0000 7fc908486640 10 monclient: build_initial_monmap 2024-03-25T18:28:35.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.267+0000 7fc908486640 10 monclient: monmap: 2024-03-25T18:28:35.268 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:35.268 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:35.268 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:35.268155+0000 2024-03-25T18:28:35.269 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:35.268155+0000 2024-03-25T18:28:35.269 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:35.269 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:35.269 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:35.269 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:35.269 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:35.269 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:35.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.267+0000 7fc908486640 1 Processor -- start 2024-03-25T18:28:35.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.267+0000 7fc908486640 1 -- start start 2024-03-25T18:28:35.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.267+0000 7fc908486640 10 monclient: init 2024-03-25T18:28:35.270 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 10 monclient(hunting): picked mon.noname-a con 0x7fc9000b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 10 monclient(hunting): picked mon.noname-c con 0x7fc9000b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 10 monclient(hunting): picked mon.noname-b con 0x7fc90014be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000af940 con 0x7fc9000b4d60 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b0a70 con 0x7fc9000b1730 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b0d50 con 0x7fc90014be90 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:35.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.268+0000 7fc908486640 10 monclient(hunting): authenticate will time out at 7830.495306s 2024-03-25T18:28:35.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.273+0000 7fc906222640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc9000b4d60 0x7fc90013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34754/0 (socket says 172.21.15.42:34754) 2024-03-25T18:28:35.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.273+0000 7fc906222640 1 -- 172.21.15.42:0/2418161552 learned_addr learned my addr 172.21.15.42:0/2418161552 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:35.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.286+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501010394 0 0) 0x7fc9000af940 con 0x7fc9000b4d60 2024-03-25T18:28:35.288 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.286+0000 7fc905220640 10 monclient(hunting): _negotiate 2024-03-25T18:28:35.288 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.286+0000 7fc905220640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:35.288 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.286+0000 7fc905220640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:35.288 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.286+0000 7fc905220640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:28:35.288 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.286+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec003520 con 0x7fc9000b4d60 2024-03-25T18:28:35.291 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.290+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490271090 0 0) 0x7fc9000b0a70 con 0x7fc9000b1730 2024-03-25T18:28:35.291 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.290+0000 7fc905220640 10 monclient(hunting): _negotiate 2024-03-25T18:28:35.292 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.290+0000 7fc905220640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:35.292 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.290+0000 7fc905220640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:35.292 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.290+0000 7fc905220640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:28:35.292 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.290+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9000af940 con 0x7fc9000b1730 2024-03-25T18:28:35.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.299+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298068540 0 0) 0x7fc9000b0d50 con 0x7fc90014be90 2024-03-25T18:28:35.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.299+0000 7fc905220640 10 monclient(hunting): _negotiate 2024-03-25T18:28:35.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.299+0000 7fc905220640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:35.300 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.299+0000 7fc905220640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:35.301 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.299+0000 7fc905220640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:28:35.301 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.299+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9000b0a70 con 0x7fc90014be90 2024-03-25T18:28:35.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909292919 0 0) 0x7fc8ec003520 con 0x7fc9000b4d60 2024-03-25T18:28:35.302 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 >> v1:172.21.15.42:6790/0 conn(0x7fc90014be90 legacy=0x7fc90014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 >> v1:172.21.15.120:6789/0 conn(0x7fc9000b1730 legacy=0x7fc9000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 10 monclient: _finish_hunting 0 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 1 monclient: found mon.noname-a 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000b1030 con 0x7fc9000b4d60 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 10 monclient: _finish_auth 0 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 10 monclient: _check_auth_tickets 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc8fc001690 con 0x7fc9000b4d60 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc905220640 10 monclient: dump: 2024-03-25T18:28:35.303 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc908486640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:28:35.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.301+0000 7fc908486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8fc0017f0 con 0x7fc9000b4d60 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc905220640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc908486640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc908486640 4 set_mon_vals no callback set 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc905220640 1 -- 172.21.15.42:0/2418161552 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc8fc001c00 con 0x7fc9000b4d60 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc905220640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc905220640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc905220640 10 monclient: dump: 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:35.315 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:35.316 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:35.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc908486640 10 monclient: shutdown 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc908486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.314+0000 7fc908486640 1 -- 172.21.15.42:0/2418161552 >> v1:172.21.15.42:6789/0 conn(0x7fc9000b4d60 legacy=0x7fc90013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.315+0000 7fc908486640 1 -- 172.21.15.42:0/2418161552 shutdown_connections 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.315+0000 7fc908486640 1 -- 172.21.15.42:0/2418161552 >> 172.21.15.42:0/2418161552 conn(0x7fc9000617a0 msgr2=0x7fc900061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.315+0000 7fc908486640 1 -- 172.21.15.42:0/2418161552 shutdown_connections 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.315+0000 7fc908486640 1 -- 172.21.15.42:0/2418161552 wait complete. 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.315+0000 7fc908486640 10 monclient: build_initial_monmap 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.315+0000 7fc908486640 10 monclient: monmap: 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:35.318 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.316+0000 7fc908486640 1 librados: starting msgr at 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.316+0000 7fc908486640 1 librados: starting objecter 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.316+0000 7fc908486640 1 Processor -- start 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.316+0000 7fc908486640 1 -- start start 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.316+0000 7fc908486640 1 librados: setting wanted keys 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.316+0000 7fc908486640 1 librados: calling monclient init 2024-03-25T18:28:35.319 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.316+0000 7fc908486640 10 monclient: init 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 10 monclient(hunting): picked mon.noname-b con 0x7fc9000b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 10 monclient(hunting): picked mon.noname-c con 0x7fc9000b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 10 monclient(hunting): picked mon.noname-a con 0x7fc90014be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900142ee0 con 0x7fc90014be90 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9001d7ab0 con 0x7fc9000b1730 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9001d86e0 con 0x7fc9000b4d60 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:35.321 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.317+0000 7fc908486640 10 monclient(hunting): authenticate will time out at 7830.543943s 2024-03-25T18:28:35.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.333+0000 7fc906222640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc9000b1730 0x7fc9001d7fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:54954/0 (socket says 172.21.15.42:54954) 2024-03-25T18:28:35.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.333+0000 7fc906222640 1 -- 172.21.15.42:0/2233655907 learned_addr learned my addr 172.21.15.42:0/2233655907 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:35.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.335+0000 7fc8f6ffd640 10 client.?.objecter ms_handle_connect 0x7fc90014be90 2024-03-25T18:28:35.337 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.335+0000 7fc8f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:35.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.338+0000 7fc8f6ffd640 10 client.?.objecter ms_handle_connect 0x7fc9000b1730 2024-03-25T18:28:35.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.338+0000 7fc8f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:35.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.340+0000 7fc8f6ffd640 10 client.?.objecter ms_handle_connect 0x7fc9000b4d60 2024-03-25T18:28:35.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.340+0000 7fc8f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:35.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.344+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348192927 0 0) 0x7fc9001d7ab0 con 0x7fc9000b1730 2024-03-25T18:28:35.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.344+0000 7fc8f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:35.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.344+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:35.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.344+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:35.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.344+0000 7fc8f6ffd640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:28:35.345 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.344+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e4004eb0 con 0x7fc9000b1730 2024-03-25T18:28:35.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.349+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199433101 0 0) 0x7fc9001d86e0 con 0x7fc9000b4d60 2024-03-25T18:28:35.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.349+0000 7fc8f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:35.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.349+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:35.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.349+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:35.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.349+0000 7fc8f6ffd640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:28:35.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.349+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9001d7ab0 con 0x7fc9000b4d60 2024-03-25T18:28:35.356 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.355+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806132824 0 0) 0x7fc900142ee0 con 0x7fc90014be90 2024-03-25T18:28:35.356 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.355+0000 7fc8f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:35.356 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.355+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:35.356 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.355+0000 7fc8f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:35.356 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.355+0000 7fc8f6ffd640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:28:35.356 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.355+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9001d86e0 con 0x7fc90014be90 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.362+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1473969465 0 0) 0x7fc8e4004eb0 con 0x7fc9000b1730 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.362+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc900142ee0 con 0x7fc9000b1730 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.362+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc8fc001bc0 con 0x7fc9000b1730 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.362+0000 7fc8f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.362+0000 7fc8f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.362+0000 7fc8f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.364 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:35.365 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:35.365 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:35.365 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:35.365 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:35.365 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:35.365 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.362+0000 7fc8f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 943402687 0 0) 0x7fc900142ee0 con 0x7fc9000b1730 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 >> v1:172.21.15.42:6790/0 conn(0x7fc9000b4d60 legacy=0x7fc900137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 >> v1:172.21.15.42:6789/0 conn(0x7fc90014be90 legacy=0x7fc9001414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 1 monclient: found mon.b 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9001d8870 con 0x7fc9000b1730 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc8f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:35.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 10 monclient: _renew_subs 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc900143900 con 0x7fc9000b1730 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 10 monclient: _renew_subs 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc900138880 con 0x7fc9000b1730 2024-03-25T18:28:35.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.367+0000 7fc908486640 1 librados: init done 2024-03-25T18:28:35.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.368+0000 7fc908486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:35.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.368+0000 7fc908486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:35.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.368+0000 7fc908486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:35.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.368+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9001d7c40 con 0x7fc9000b1730 2024-03-25T18:28:35.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.647+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8fc0011f0 con 0x7fc9000b1730 2024-03-25T18:28:35.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.647+0000 7fc8f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:35.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.647+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fc8fc004c30 con 0x7fc9000b1730 2024-03-25T18:28:35.648 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.647+0000 7fc8f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.647+0000 7fc8f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.647+0000 7fc8f6ffd640 10 monclient: dump: 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:35.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.647+0000 7fc904a1f640 4 set_mon_vals no callback set 2024-03-25T18:28:35.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.648+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fc8fc01ac80 con 0x7fc9000b1730 2024-03-25T18:28:35.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.648+0000 7fc905a21640 10 monclient: get_auth_request con 0x7fc8e406dd50 auth_method 0 2024-03-25T18:28:35.650 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.649+0000 7fc8f6ffd640 10 client.4639.objecter ms_handle_connect 0x7fc8e406dd50 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fc8fc0549d0 con 0x7fc9000b1730 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 10 client.4639.objecter ms_dispatch 0x7fc900000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8fc050310 con 0x7fc9000b1730 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:35.655 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.653+0000 7fc8f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:35.907 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.906+0000 7fc908486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:35.907 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.906+0000 7fc908486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:35.907 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.906+0000 7fc908486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:35.908 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.906+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9000627b0 con 0x7fc9000b1730 2024-03-25T18:28:35.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.953+0000 7fc8f6ffd640 1 -- 172.21.15.42:0/2233655907 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fc8fc052410 con 0x7fc9000b1730 2024-03-25T18:28:35.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.953+0000 7fc8f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:35.954 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.953+0000 7fc8f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:35.955 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:35.955 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 10 librados: watch_flush enter 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 10 librados: watch_flush exit 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 10 client.4639.objecter successfully canceled tick 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 >> v1:172.21.15.42:6800/3081750888 conn(0x7fc8e406dd50 legacy=0x7fc8e4070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 10 monclient: shutdown 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 >> v1:172.21.15.120:6789/0 conn(0x7fc9000b1730 legacy=0x7fc9001d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 shutdown_connections 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 >> 172.21.15.42:0/2233655907 conn(0x7fc9000617a0 msgr2=0x7fc90013c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:35.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 shutdown_connections 2024-03-25T18:28:35.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 1 -- 172.21.15.42:0/2233655907 wait complete. 2024-03-25T18:28:35.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:35.955+0000 7fc908486640 1 librados: shutdown 2024-03-25T18:28:35.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:42.775 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:42.892 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.891+0000 7f0527255640 10 monclient: get_monmap_and_config 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.891+0000 7f0527255640 10 monclient: build_initial_monmap 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.891+0000 7f0527255640 10 monclient: monmap: 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:42.892440+0000 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:42.892440+0000 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:42.893 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:42.894 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 1 Processor -- start 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 1 -- start start 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 10 monclient: init 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 10 monclient(hunting): picked mon.noname-a con 0x7f05200a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 10 monclient(hunting): picked mon.noname-b con 0x7f052013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 10 monclient(hunting): picked mon.noname-c con 0x7f05201430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0520142520 con 0x7f05200a98f0 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05200b28a0 con 0x7f05201430c0 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05200b2a30 con 0x7f052013f710 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.892+0000 7f0527255640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:42.895 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.893+0000 7f0527255640 10 monclient(hunting): authenticate will time out at 7838.119552s 2024-03-25T18:28:42.900 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.899+0000 7f05257f2640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f05201430c0 0x7f052014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:47142/0 (socket says 172.21.15.42:47142) 2024-03-25T18:28:42.901 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.899+0000 7f05257f2640 1 -- 172.21.15.42:0/1133377581 learned_addr learned my addr 172.21.15.42:0/1133377581 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:42.912 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.911+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618049205 0 0) 0x7f05200b28a0 con 0x7f05201430c0 2024-03-25T18:28:42.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.911+0000 7f051f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:42.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.911+0000 7f051f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:42.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.911+0000 7f051f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:42.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.911+0000 7f051f7fe640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:28:42.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.911+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510003520 con 0x7f05201430c0 2024-03-25T18:28:42.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.912+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960481128 0 0) 0x7f0520142520 con 0x7f05200a98f0 2024-03-25T18:28:42.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.912+0000 7f051f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:42.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.912+0000 7f051f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:42.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.912+0000 7f051f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:42.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.912+0000 7f051f7fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:28:42.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.912+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05200b28a0 con 0x7f05200a98f0 2024-03-25T18:28:42.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.921+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721147803 0 0) 0x7f0510003520 con 0x7f05201430c0 2024-03-25T18:28:42.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.921+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 >> v1:172.21.15.42:6790/0 conn(0x7f052013f710 legacy=0x7f05201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:28:42.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 >> v1:172.21.15.42:6789/0 conn(0x7f05200a98f0 legacy=0x7f05200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:42.923 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05200b44c0 con 0x7f05201430c0 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0518001a10 con 0x7f05201430c0 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f051f7fe640 10 monclient: dump: 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:42.924 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f0527255640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:28:42.925 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.922+0000 7f0527255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0518001b70 con 0x7f05201430c0 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f051f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f051f7fe640 1 -- 172.21.15.42:0/1133377581 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0518001f80 con 0x7f05201430c0 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f051f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f051f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f051f7fe640 10 monclient: dump: 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:42.931 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f0527255640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:42.932 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.930+0000 7f0527255640 4 set_mon_vals no callback set 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.931+0000 7f0527255640 10 monclient: shutdown 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.931+0000 7f0527255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.931+0000 7f0527255640 1 -- 172.21.15.42:0/1133377581 >> v1:172.21.15.120:6789/0 conn(0x7f05201430c0 legacy=0x7f052014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.931+0000 7f0527255640 1 -- 172.21.15.42:0/1133377581 shutdown_connections 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.931+0000 7f0527255640 1 -- 172.21.15.42:0/1133377581 >> 172.21.15.42:0/1133377581 conn(0x7f05200b7ea0 msgr2=0x7f05200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.931+0000 7f0527255640 1 -- 172.21.15.42:0/1133377581 shutdown_connections 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.931+0000 7f0527255640 1 -- 172.21.15.42:0/1133377581 wait complete. 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 10 monclient: build_initial_monmap 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 10 monclient: monmap: 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:42.934 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 1 librados: starting msgr at 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 1 librados: starting objecter 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 1 Processor -- start 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 1 -- start start 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 1 librados: setting wanted keys 2024-03-25T18:28:42.935 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 1 librados: calling monclient init 2024-03-25T18:28:42.936 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.932+0000 7f0527255640 10 monclient: init 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.933+0000 7f0527255640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.933+0000 7f0527255640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.933+0000 7f0527255640 10 monclient(hunting): picked mon.noname-b con 0x7f05200a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.933+0000 7f0527255640 10 monclient(hunting): picked mon.noname-a con 0x7f052013f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.934+0000 7f0527255640 10 monclient(hunting): picked mon.noname-c con 0x7f05201430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.934+0000 7f0527255640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05201d7870 con 0x7f052013f710 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.934+0000 7f0527255640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05201d7490 con 0x7f05200a98f0 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.934+0000 7f0527255640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05201d7620 con 0x7f05201430c0 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.934+0000 7f0527255640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:42.937 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.934+0000 7f0527255640 10 monclient(hunting): authenticate will time out at 7838.160735s 2024-03-25T18:28:42.945 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.944+0000 7f0524ff1640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f05200a98f0 0x7f0520135110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:47170/0 (socket says 172.21.15.42:47170) 2024-03-25T18:28:42.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.944+0000 7f0524ff1640 1 -- 172.21.15.42:0/3387985379 learned_addr learned my addr 172.21.15.42:0/3387985379 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:42.948 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.947+0000 7f051d7fa640 10 client.?.objecter ms_handle_connect 0x7f05200a98f0 2024-03-25T18:28:42.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.947+0000 7f051d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:42.951 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.950+0000 7f051d7fa640 10 client.?.objecter ms_handle_connect 0x7f052013f710 2024-03-25T18:28:42.951 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.950+0000 7f051d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:42.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.958+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442342629 0 0) 0x7f05201d7490 con 0x7f05200a98f0 2024-03-25T18:28:42.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.958+0000 7f051d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:42.959 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.958+0000 7f051d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:42.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.958+0000 7f051d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:42.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.958+0000 7f051d7fa640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:28:42.960 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.958+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504004eb0 con 0x7f05200a98f0 2024-03-25T18:28:42.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683022556 0 0) 0x7f05201d7870 con 0x7f052013f710 2024-03-25T18:28:42.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:42.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:42.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:42.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:28:42.964 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05201d7490 con 0x7f052013f710 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 10 client.?.objecter ms_handle_connect 0x7f05201430c0 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.963+0000 7f051d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.964+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3731799679 0 0) 0x7f0504004eb0 con 0x7f05200a98f0 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.964+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05201d7870 con 0x7f05200a98f0 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.964+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0514001a10 con 0x7f05200a98f0 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.964+0000 7f051d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.964+0000 7f051d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.964+0000 7f051d7fa640 10 monclient(hunting): dump: 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:42.965 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:42.966 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.964+0000 7f051d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:42.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1027990042 0 0) 0x7f05201d7870 con 0x7f05200a98f0 2024-03-25T18:28:42.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 >> v1:172.21.15.42:6790/0 conn(0x7f05201430c0 legacy=0x7f05201386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 >> v1:172.21.15.42:6789/0 conn(0x7f052013f710 legacy=0x7f05201d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:42.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 1 monclient: found mon.b 2024-03-25T18:28:42.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520138df0 con 0x7f05200a98f0 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f051d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 10 monclient: _renew_subs 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0520139040 con 0x7f05200a98f0 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 10 monclient: _renew_subs 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05201381b0 con 0x7f05200a98f0 2024-03-25T18:28:42.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.981+0000 7f0527255640 1 librados: init done 2024-03-25T18:28:42.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.983+0000 7f0527255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:42.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.983+0000 7f0527255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:42.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.983+0000 7f0527255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:42.985 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:42.983+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05200627b0 con 0x7f05200a98f0 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.279+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0514001c50 con 0x7f05200a98f0 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.279+0000 7f051d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.279+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f05140038c0 con 0x7f05200a98f0 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.279+0000 7f051d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.279+0000 7f051d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.279+0000 7f051d7fa640 10 monclient: dump: 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:43.281 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:43.282 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:43.282 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:43.282 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:43.282 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:43.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.279+0000 7f051effd640 4 set_mon_vals no callback set 2024-03-25T18:28:43.282 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.280+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f0514003ae0 con 0x7f05200a98f0 2024-03-25T18:28:43.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.281+0000 7f051ffff640 10 monclient: get_auth_request con 0x7f050406dd10 auth_method 0 2024-03-25T18:28:43.283 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.281+0000 7f051d7fa640 10 client.4645.objecter ms_handle_connect 0x7f050406dd10 2024-03-25T18:28:43.286 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f0514054700 con 0x7f05200a98f0 2024-03-25T18:28:43.286 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 10 client.4645.objecter ms_dispatch 0x7f0520000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:43.286 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:43.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:43.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T18:28:43.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0514050210 con 0x7f05200a98f0 2024-03-25T18:28:43.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:43.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.285+0000 7f051d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:43.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.510+0000 7f0527255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:43.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.510+0000 7f0527255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:43.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.510+0000 7f0527255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:43.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.510+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0520069f40 con 0x7f05200a98f0 2024-03-25T18:28:43.530 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.529+0000 7f051d7fa640 1 -- 172.21.15.42:0/3387985379 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f0514052140 con 0x7f05200a98f0 2024-03-25T18:28:43.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.529+0000 7f051d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:43.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.529+0000 7f051d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:43.531 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:43.531 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:43.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 10 librados: watch_flush enter 2024-03-25T18:28:43.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 10 librados: watch_flush exit 2024-03-25T18:28:43.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T18:28:43.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 10 client.4645.objecter successfully canceled tick 2024-03-25T18:28:43.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 >> v1:172.21.15.42:6800/3081750888 conn(0x7f050406dd10 legacy=0x7f05040701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 10 monclient: shutdown 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 >> v1:172.21.15.120:6789/0 conn(0x7f05200a98f0 legacy=0x7f0520135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 shutdown_connections 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 >> 172.21.15.42:0/3387985379 conn(0x7f05200b7ea0 msgr2=0x7f0520074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.531+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 shutdown_connections 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.532+0000 7f0527255640 1 -- 172.21.15.42:0/3387985379 wait complete. 2024-03-25T18:28:43.534 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:43.532+0000 7f0527255640 1 librados: shutdown 2024-03-25T18:28:43.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:50.353 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:50.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.470+0000 7f72b417b640 10 monclient: get_monmap_and_config 2024-03-25T18:28:50.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.470+0000 7f72b417b640 10 monclient: build_initial_monmap 2024-03-25T18:28:50.472 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.470+0000 7f72b417b640 10 monclient: monmap: 2024-03-25T18:28:50.472 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:50.472 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:50.472 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:50.472003+0000 2024-03-25T18:28:50.472 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:50.472003+0000 2024-03-25T18:28:50.473 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:50.473 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:50.473 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:50.473 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:50.473 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:50.473 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:50.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.471+0000 7f72b417b640 1 Processor -- start 2024-03-25T18:28:50.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.471+0000 7f72b417b640 1 -- start start 2024-03-25T18:28:50.474 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.471+0000 7f72b417b640 10 monclient: init 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 10 monclient(hunting): picked mon.noname-b con 0x7f72ac0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 10 monclient(hunting): picked mon.noname-a con 0x7f72ac13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 10 monclient(hunting): picked mon.noname-c con 0x7f72ac1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac142520 con 0x7f72ac13f710 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0b28a0 con 0x7f72ac1430c0 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0b2a30 con 0x7f72ac0a98f0 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:50.475 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.472+0000 7f72b417b640 10 monclient(hunting): authenticate will time out at 7845.699464s 2024-03-25T18:28:50.477 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.476+0000 7f72b1f17640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f72ac0a98f0 0x7f72ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:42652/0 (socket says 172.21.15.42:42652) 2024-03-25T18:28:50.478 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.476+0000 7f72b1f17640 1 -- 172.21.15.42:0/1698280610 learned_addr learned my addr 172.21.15.42:0/1698280610 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:50.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.485+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172723613 0 0) 0x7f72ac0b2a30 con 0x7f72ac0a98f0 2024-03-25T18:28:50.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.485+0000 7f72b0f15640 10 monclient(hunting): _negotiate 2024-03-25T18:28:50.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.485+0000 7f72b0f15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:50.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.485+0000 7f72b0f15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:50.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.485+0000 7f72b0f15640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:28:50.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.485+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7298003520 con 0x7f72ac0a98f0 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276686179 0 0) 0x7f7298003520 con 0x7f72ac0a98f0 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 >> v1:172.21.15.120:6789/0 conn(0x7f72ac1430c0 legacy=0x7f72ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 >> v1:172.21.15.42:6789/0 conn(0x7f72ac13f710 legacy=0x7f72ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 10 monclient: _finish_hunting 0 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 1 monclient: found mon.noname-b 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:28:50.499 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac0b44c0 con 0x7f72ac0a98f0 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 10 monclient: _finish_auth 0 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 10 monclient: _check_auth_tickets 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f72a8001690 con 0x7f72ac0a98f0 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b0f15640 10 monclient: dump: 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:50.500 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:50.501 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:50.501 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:50.501 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b417b640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:28:50.501 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.497+0000 7f72b417b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:50.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72a8001860 con 0x7f72ac0a98f0 2024-03-25T18:28:50.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b0f15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:50.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b0f15640 1 -- 172.21.15.42:0/1698280610 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f72a8001cb0 con 0x7f72ac0a98f0 2024-03-25T18:28:50.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b417b640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b417b640 4 set_mon_vals no callback set 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b0f15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b0f15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b0f15640 10 monclient: dump: 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:50.513 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b417b640 10 monclient: shutdown 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b417b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b417b640 1 -- 172.21.15.42:0/1698280610 >> v1:172.21.15.42:6790/0 conn(0x7f72ac0a98f0 legacy=0x7f72ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b417b640 1 -- 172.21.15.42:0/1698280610 shutdown_connections 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.511+0000 7f72b417b640 1 -- 172.21.15.42:0/1698280610 >> 172.21.15.42:0/1698280610 conn(0x7f72ac0b7ea0 msgr2=0x7f72ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.512+0000 7f72b417b640 1 -- 172.21.15.42:0/1698280610 shutdown_connections 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.512+0000 7f72b417b640 1 -- 172.21.15.42:0/1698280610 wait complete. 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 10 monclient: build_initial_monmap 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 10 monclient: monmap: 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:50.516 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 1 librados: starting msgr at 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 1 librados: starting objecter 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 1 Processor -- start 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 1 -- start start 2024-03-25T18:28:50.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 1 librados: setting wanted keys 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 1 librados: calling monclient init 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.513+0000 7f72b417b640 10 monclient: init 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 10 monclient(hunting): picked mon.noname-b con 0x7f72ac0a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 10 monclient(hunting): picked mon.noname-c con 0x7f72ac13f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 10 monclient(hunting): picked mon.noname-a con 0x7f72ac1430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac1db410 con 0x7f72ac1430c0 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0ab950 con 0x7f72ac0a98f0 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0b0df0 con 0x7f72ac13f710 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:50.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.514+0000 7f72b417b640 10 monclient(hunting): authenticate will time out at 7845.740959s 2024-03-25T18:28:50.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.526+0000 7f72b1716640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f72ac13f710 0x7f72ac0b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:42716/0 (socket says 172.21.15.42:42716) 2024-03-25T18:28:50.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.526+0000 7f72b1716640 1 -- 172.21.15.42:0/1495034454 learned_addr learned my addr 172.21.15.42:0/1495034454 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:50.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.526+0000 7f72a27fc640 10 client.?.objecter ms_handle_connect 0x7f72ac13f710 2024-03-25T18:28:50.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.526+0000 7f72a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:50.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.532+0000 7f72a27fc640 10 client.?.objecter ms_handle_connect 0x7f72ac0a98f0 2024-03-25T18:28:50.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.532+0000 7f72a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:50.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.540+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626036053 0 0) 0x7f72ac0ab950 con 0x7f72ac0a98f0 2024-03-25T18:28:50.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.540+0000 7f72a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:50.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.540+0000 7f72a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:50.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.540+0000 7f72a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:50.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.540+0000 7f72a27fc640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:28:50.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.540+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7290004eb0 con 0x7f72ac0a98f0 2024-03-25T18:28:50.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.541+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244332256 0 0) 0x7f72ac0b0df0 con 0x7f72ac13f710 2024-03-25T18:28:50.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.541+0000 7f72a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:50.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.541+0000 7f72a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:50.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.541+0000 7f72a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:50.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.541+0000 7f72a27fc640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:28:50.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.541+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac0ab950 con 0x7f72ac13f710 2024-03-25T18:28:50.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.560+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 125385674 0 0) 0x7f72ac0ab950 con 0x7f72ac13f710 2024-03-25T18:28:50.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.560+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72ac0b0df0 con 0x7f72ac13f710 2024-03-25T18:28:50.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.560+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f729c0019f0 con 0x7f72ac13f710 2024-03-25T18:28:50.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.560+0000 7f72a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:50.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.560+0000 7f72a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:50.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.560+0000 7f72a27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:50.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.560+0000 7f72a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.566+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2977139082 0 0) 0x7f7290004eb0 con 0x7f72ac0a98f0 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.566+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72ac0ab950 con 0x7f72ac0a98f0 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.566+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f72a8001b70 con 0x7f72ac0a98f0 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.566+0000 7f72a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.566+0000 7f72a27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.566+0000 7f72a27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:50.568 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:50.569 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:50.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1848724032 0 0) 0x7f72ac0b0df0 con 0x7f72ac13f710 2024-03-25T18:28:50.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 >> v1:172.21.15.120:6789/0 conn(0x7f72ac0a98f0 legacy=0x7f72ac1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 >> v1:172.21.15.42:6789/0 conn(0x7f72ac1430c0 legacy=0x7f72ac1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 1 monclient: found mon.c 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac0b0f80 con 0x7f72ac13f710 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 5 monclient: authenticate success, global_id 4643 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 10 monclient: _renew_subs 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:28:50.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac0b07f0 con 0x7f72ac13f710 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 10 monclient: _renew_subs 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac1d48f0 con 0x7f72ac13f710 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.567+0000 7f72b417b640 1 librados: init done 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.568+0000 7f72b417b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.568+0000 7f72b417b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.568+0000 7f72b417b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:28:50.572 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.568+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac0b1190 con 0x7f72ac13f710 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.882+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f729c003610 con 0x7f72ac13f710 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.882+0000 7f72a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.882+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f729c0039e0 con 0x7f72ac13f710 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.882+0000 7f72a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.882+0000 7f72a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.882+0000 7f72a27fc640 10 monclient: dump: 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.884 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:50.885 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:50.885 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:50.885 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:50.885 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:50.885 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:50.885 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:50.885 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.882+0000 7f72a3fff640 4 set_mon_vals no callback set 2024-03-25T18:28:50.886 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.883+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f729c003c00 con 0x7f72ac13f710 2024-03-25T18:28:50.886 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.883+0000 7f72b1f17640 10 monclient: get_auth_request con 0x7f729006ddf0 auth_method 0 2024-03-25T18:28:50.886 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.884+0000 7f72a27fc640 10 client.4643.objecter ms_handle_connect 0x7f729006ddf0 2024-03-25T18:28:50.892 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f729c054860 con 0x7f72ac13f710 2024-03-25T18:28:50.892 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 10 client.4643.objecter ms_dispatch 0x7f72ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:50.892 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:50.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:50.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T18:28:50.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f729c050370 con 0x7f72ac13f710 2024-03-25T18:28:50.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:50.893 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:50.891+0000 7f72a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:51.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.139+0000 7f72b417b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:51.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.139+0000 7f72b417b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:51.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.139+0000 7f72b417b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:28:51.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.139+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72ac0627b0 con 0x7f72ac13f710 2024-03-25T18:28:51.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.187+0000 7f72a27fc640 1 -- 172.21.15.42:0/1495034454 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f729c0522a0 con 0x7f72ac13f710 2024-03-25T18:28:51.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.188+0000 7f72a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:51.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.188+0000 7f72a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:51.189 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:51.190 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:51.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 10 librados: watch_flush enter 2024-03-25T18:28:51.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 10 librados: watch_flush exit 2024-03-25T18:28:51.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T18:28:51.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 10 client.4643.objecter successfully canceled tick 2024-03-25T18:28:51.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 >> v1:172.21.15.42:6800/3081750888 conn(0x7f729006ddf0 legacy=0x7f72900702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:51.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 10 monclient: shutdown 2024-03-25T18:28:51.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:51.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.190+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 >> v1:172.21.15.42:6790/0 conn(0x7f72ac13f710 legacy=0x7f72ac0b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:51.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.191+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 shutdown_connections 2024-03-25T18:28:51.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.191+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 >> 172.21.15.42:0/1495034454 conn(0x7f72ac0b7ea0 msgr2=0x7f72ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:51.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.191+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 shutdown_connections 2024-03-25T18:28:51.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.191+0000 7f72b417b640 1 -- 172.21.15.42:0/1495034454 wait complete. 2024-03-25T18:28:51.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:51.191+0000 7f72b417b640 1 librados: shutdown 2024-03-25T18:28:51.204 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:53.212 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:36.381 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.263 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:36.432 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.298 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:36.468 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.315 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:36.435 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.322 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:36.443 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.328 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:36.498 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.344 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:36.513 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.348 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:36.468 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:53.367 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:36.537 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:58.013 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.125+0000 7f25e584d640 10 monclient: get_monmap_and_config 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.125+0000 7f25e584d640 10 monclient: build_initial_monmap 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.125+0000 7f25e584d640 10 monclient: monmap: 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:28:58.126686+0000 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:28:58.126686+0000 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:58.127 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:58.128 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:28:58.128 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:28:58.128 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.126+0000 7f25e584d640 1 Processor -- start 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.126+0000 7f25e584d640 1 -- start start 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.126+0000 7f25e584d640 10 monclient: init 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 10 monclient(hunting): picked mon.noname-a con 0x7f25e00a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 10 monclient(hunting): picked mon.noname-b con 0x7f25e013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 10 monclient(hunting): picked mon.noname-c con 0x7f25e01430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e0142520 con 0x7f25e00a98f0 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e00b28a0 con 0x7f25e01430c0 2024-03-25T18:28:58.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e00b2a30 con 0x7f25e013f710 2024-03-25T18:28:58.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:58.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.127+0000 7f25e584d640 10 monclient(hunting): authenticate will time out at 7853.354337s 2024-03-25T18:28:58.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.154+0000 7f25df7fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f25e01430c0 0x7f25e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:53006/0 (socket says 172.21.15.42:53006) 2024-03-25T18:28:58.155 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.154+0000 7f25df7fe640 1 -- 172.21.15.42:0/341788946 learned_addr learned my addr 172.21.15.42:0/341788946 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:58.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.167+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404774517 0 0) 0x7f25e00b28a0 con 0x7f25e01430c0 2024-03-25T18:28:58.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.167+0000 7f25ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:58.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.167+0000 7f25ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:58.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.167+0000 7f25ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:58.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.167+0000 7f25ddffb640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:28:58.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.167+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c8003520 con 0x7f25e01430c0 2024-03-25T18:28:58.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.170+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701789007 0 0) 0x7f25e00b2a30 con 0x7f25e013f710 2024-03-25T18:28:58.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.170+0000 7f25ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:58.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.170+0000 7f25ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:58.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.170+0000 7f25ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:58.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.170+0000 7f25ddffb640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:28:58.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.170+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e00b28a0 con 0x7f25e013f710 2024-03-25T18:28:58.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953102935 0 0) 0x7f25c8003520 con 0x7f25e01430c0 2024-03-25T18:28:58.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 >> v1:172.21.15.42:6790/0 conn(0x7f25e013f710 legacy=0x7f25e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:58.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 >> v1:172.21.15.42:6789/0 conn(0x7f25e00a98f0 legacy=0x7f25e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:58.182 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 1 monclient: found mon.noname-c 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e00b44c0 con 0x7f25e01430c0 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f25d8001a10 con 0x7f25e01430c0 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25e584d640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25e584d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.180+0000 7f25ddffb640 10 monclient: dump: 2024-03-25T18:28:58.183 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:58.184 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.181+0000 7f25e584d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d8001b70 con 0x7f25e01430c0 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25ddffb640 1 -- 172.21.15.42:0/341788946 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f25d8001f80 con 0x7f25e01430c0 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25e584d640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25e584d640 4 set_mon_vals no callback set 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25ddffb640 10 monclient: dump: 2024-03-25T18:28:58.190 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:58.191 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:58.191 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.191 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.191 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:58.191 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:58.191 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:58.192 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:58.192 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:58.192 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:58.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25e584d640 10 monclient: shutdown 2024-03-25T18:28:58.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25e584d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:58.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.189+0000 7f25e584d640 1 -- 172.21.15.42:0/341788946 >> v1:172.21.15.120:6789/0 conn(0x7f25e01430c0 legacy=0x7f25e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:58.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.190+0000 7f25e584d640 1 -- 172.21.15.42:0/341788946 shutdown_connections 2024-03-25T18:28:58.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.190+0000 7f25e584d640 1 -- 172.21.15.42:0/341788946 >> 172.21.15.42:0/341788946 conn(0x7f25e00b7ea0 msgr2=0x7f25e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:58.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.190+0000 7f25e584d640 1 -- 172.21.15.42:0/341788946 shutdown_connections 2024-03-25T18:28:58.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.190+0000 7f25e584d640 1 -- 172.21.15.42:0/341788946 wait complete. 2024-03-25T18:28:58.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.190+0000 7f25e584d640 10 monclient: build_initial_monmap 2024-03-25T18:28:58.195 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.190+0000 7f25e584d640 10 monclient: monmap: 2024-03-25T18:28:58.195 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:28:58.197 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:58.197 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:28:58.197 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.191+0000 7f25e584d640 1 librados: starting msgr at 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.191+0000 7f25e584d640 1 librados: starting objecter 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.191+0000 7f25e584d640 1 Processor -- start 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.191+0000 7f25e584d640 1 -- start start 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.191+0000 7f25e584d640 1 librados: setting wanted keys 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.191+0000 7f25e584d640 1 librados: calling monclient init 2024-03-25T18:28:58.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.191+0000 7f25e584d640 10 monclient: init 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.192+0000 7f25e584d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.192+0000 7f25e584d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.192+0000 7f25e584d640 10 monclient(hunting): picked mon.noname-b con 0x7f25e00a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.192+0000 7f25e584d640 10 monclient(hunting): picked mon.noname-a con 0x7f25e013f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.192+0000 7f25e584d640 10 monclient(hunting): picked mon.noname-c con 0x7f25e01430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.192+0000 7f25e584d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e01d7500 con 0x7f25e013f710 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.193+0000 7f25e584d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e01394a0 con 0x7f25e00a98f0 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.193+0000 7f25e584d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e0138b50 con 0x7f25e01430c0 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.193+0000 7f25e584d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:58.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.193+0000 7f25e584d640 10 monclient(hunting): authenticate will time out at 7853.419633s 2024-03-25T18:28:58.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.198+0000 7f25de7fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25e013f710 0x7f25e01d7c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41618/0 (socket says 172.21.15.42:41618) 2024-03-25T18:28:58.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.198+0000 7f25de7fc640 1 -- 172.21.15.42:0/1904827034 learned_addr learned my addr 172.21.15.42:0/1904827034 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:28:58.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.200+0000 7f25c7fff640 10 client.?.objecter ms_handle_connect 0x7f25e00a98f0 2024-03-25T18:28:58.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.200+0000 7f25c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:58.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.204+0000 7f25c7fff640 10 client.?.objecter ms_handle_connect 0x7f25e013f710 2024-03-25T18:28:58.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.204+0000 7f25c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:58.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.207+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304116035 0 0) 0x7f25e01394a0 con 0x7f25e00a98f0 2024-03-25T18:28:58.208 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.207+0000 7f25c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:58.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.207+0000 7f25c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:58.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.207+0000 7f25c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:58.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.207+0000 7f25c7fff640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:28:58.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.207+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc004eb0 con 0x7f25e00a98f0 2024-03-25T18:28:58.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.211+0000 7f25c7fff640 10 client.?.objecter ms_handle_connect 0x7f25e01430c0 2024-03-25T18:28:58.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.211+0000 7f25c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:58.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.214+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120995839 0 0) 0x7f25e01d7500 con 0x7f25e013f710 2024-03-25T18:28:58.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.214+0000 7f25c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:58.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.214+0000 7f25c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:58.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.214+0000 7f25c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:58.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.214+0000 7f25c7fff640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:28:58.216 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.214+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e01394a0 con 0x7f25e013f710 2024-03-25T18:28:58.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.225+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444380383 0 0) 0x7f25cc004eb0 con 0x7f25e00a98f0 2024-03-25T18:28:58.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.225+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25e01d7500 con 0x7f25e00a98f0 2024-03-25T18:28:58.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.225+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f25d00018c0 con 0x7f25e00a98f0 2024-03-25T18:28:58.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.225+0000 7f25c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:58.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.225+0000 7f25c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:58.227 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.225+0000 7f25c7fff640 10 monclient(hunting): dump: 2024-03-25T18:28:58.227 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:58.228 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.225+0000 7f25c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:58.233 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.232+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091244591 0 0) 0x7f25e0138b50 con 0x7f25e01430c0 2024-03-25T18:28:58.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.232+0000 7f25c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:58.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.232+0000 7f25c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:58.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.232+0000 7f25c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:58.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.232+0000 7f25c7fff640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:28:58.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.232+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc004eb0 con 0x7f25e01430c0 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.234+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300965265 0 0) 0x7f25e01394a0 con 0x7f25e013f710 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.234+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25e0138b50 con 0x7f25e013f710 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.234+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f25d40019f0 con 0x7f25e013f710 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.234+0000 7f25c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.234+0000 7f25c7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.234+0000 7f25c7fff640 10 monclient(hunting): dump: 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.235 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.236 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:58.236 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:58.236 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:58.236 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:58.236 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:58.236 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1171914118 0 0) 0x7f25e01d7500 con 0x7f25e00a98f0 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 >> v1:172.21.15.42:6790/0 conn(0x7f25e01430c0 legacy=0x7f25e0134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 >> v1:172.21.15.42:6789/0 conn(0x7f25e013f710 legacy=0x7f25e01d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 1 monclient: found mon.b 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e01d8380 con 0x7f25e00a98f0 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:58.241 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 10 monclient: _renew_subs 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e0139710 con 0x7f25e00a98f0 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 10 monclient: _renew_subs 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e01d7750 con 0x7f25e00a98f0 2024-03-25T18:28:58.242 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.239+0000 7f25e584d640 1 librados: init done 2024-03-25T18:28:58.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.240+0000 7f25e584d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:58.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.240+0000 7f25e584d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:58.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.240+0000 7f25e584d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:58.243 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.240+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e00627b0 con 0x7f25e00a98f0 2024-03-25T18:28:58.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.509+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d00035f0 con 0x7f25e00a98f0 2024-03-25T18:28:58.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.509+0000 7f25c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:58.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.509+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f25d0003900 con 0x7f25e00a98f0 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.509+0000 7f25c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.509+0000 7f25c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.509+0000 7f25c7fff640 10 monclient: dump: 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:28:58.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.509+0000 7f25dd7fa640 4 set_mon_vals no callback set 2024-03-25T18:28:58.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.510+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f25d00198d0 con 0x7f25e00a98f0 2024-03-25T18:28:58.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.510+0000 7f25de7fc640 10 monclient: get_auth_request con 0x7f25cc06df20 auth_method 0 2024-03-25T18:28:58.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.510+0000 7f25c7fff640 10 client.4654.objecter ms_handle_connect 0x7f25cc06df20 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f25d0053600 con 0x7f25e00a98f0 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 10 client.4654.objecter ms_dispatch 0x7f25e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 3 client.4654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 3 client.4654.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25d004ef40 con 0x7f25e00a98f0 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:58.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.515+0000 7f25c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:58.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.765+0000 7f25e584d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:58.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.765+0000 7f25e584d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:58.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.765+0000 7f25e584d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:28:58.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.765+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25e0069f40 con 0x7f25e00a98f0 2024-03-25T18:28:58.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.786+0000 7f25c7fff640 1 -- 172.21.15.42:0/1904827034 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f25d0051040 con 0x7f25e00a98f0 2024-03-25T18:28:58.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.786+0000 7f25c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:58.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.786+0000 7f25c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:58.788 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:28:58.788 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:58.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.788+0000 7f25e584d640 10 librados: watch_flush enter 2024-03-25T18:28:58.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.788+0000 7f25e584d640 10 librados: watch_flush exit 2024-03-25T18:28:58.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.788+0000 7f25e584d640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T18:28:58.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.788+0000 7f25e584d640 10 client.4654.objecter successfully canceled tick 2024-03-25T18:28:58.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 >> v1:172.21.15.42:6800/3081750888 conn(0x7f25cc06df20 legacy=0x7f25cc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:58.791 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 10 monclient: shutdown 2024-03-25T18:28:58.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:58.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 >> v1:172.21.15.120:6789/0 conn(0x7f25e00a98f0 legacy=0x7f25e0138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:58.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 shutdown_connections 2024-03-25T18:28:58.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 >> 172.21.15.42:0/1904827034 conn(0x7f25e00b7ea0 msgr2=0x7f25e013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:58.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 shutdown_connections 2024-03-25T18:28:58.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 1 -- 172.21.15.42:0/1904827034 wait complete. 2024-03-25T18:28:58.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:28:58.789+0000 7f25e584d640 1 librados: shutdown 2024-03-25T18:28:58.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:05.610 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:05.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.725+0000 7efca3c63640 10 monclient: get_monmap_and_config 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.725+0000 7efca3c63640 10 monclient: build_initial_monmap 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.725+0000 7efca3c63640 10 monclient: monmap: 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:05.726410+0000 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:05.726410+0000 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:05.727 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.726+0000 7efca3c63640 1 Processor -- start 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.726+0000 7efca3c63640 1 -- start start 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.726+0000 7efca3c63640 10 monclient: init 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 10 monclient(hunting): picked mon.noname-a con 0x7efc9c0a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 10 monclient(hunting): picked mon.noname-c con 0x7efc9c13f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 10 monclient(hunting): picked mon.noname-b con 0x7efc9c1430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c142520 con 0x7efc9c0a98f0 2024-03-25T18:29:05.729 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c0b28a0 con 0x7efc9c13f710 2024-03-25T18:29:05.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c0b2a30 con 0x7efc9c1430c0 2024-03-25T18:29:05.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:05.730 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.727+0000 7efca3c63640 10 monclient(hunting): authenticate will time out at 7860.954012s 2024-03-25T18:29:05.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.730+0000 7efca11fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7efc9c13f710 0x7efc9c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:57606/0 (socket says 172.21.15.42:57606) 2024-03-25T18:29:05.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.730+0000 7efca11fe640 1 -- 172.21.15.42:0/2711110009 learned_addr learned my addr 172.21.15.42:0/2711110009 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:05.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.739+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668386308 0 0) 0x7efc9c0b28a0 con 0x7efc9c13f710 2024-03-25T18:29:05.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.739+0000 7efca09fd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:05.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.739+0000 7efca09fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:05.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.739+0000 7efca09fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:05.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.739+0000 7efca09fd640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:29:05.741 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.739+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc88003520 con 0x7efc9c13f710 2024-03-25T18:29:05.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.749+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908070316 0 0) 0x7efc9c142520 con 0x7efc9c0a98f0 2024-03-25T18:29:05.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.749+0000 7efca09fd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:05.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.749+0000 7efca09fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:05.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.749+0000 7efca09fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:05.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.749+0000 7efca09fd640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:29:05.751 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.749+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c0b28a0 con 0x7efc9c0a98f0 2024-03-25T18:29:05.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1811845248 0 0) 0x7efc88003520 con 0x7efc9c13f710 2024-03-25T18:29:05.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 >> v1:172.21.15.42:6790/0 conn(0x7efc9c1430c0 legacy=0x7efc9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 >> v1:172.21.15.42:6789/0 conn(0x7efc9c0a98f0 legacy=0x7efc9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 1 monclient: found mon.noname-c 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c0b44c0 con 0x7efc9c13f710 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 10 monclient: _finish_auth 0 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7efc8c001690 con 0x7efc9c13f710 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca09fd640 10 monclient: dump: 2024-03-25T18:29:05.758 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca3c63640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:29:05.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.756+0000 7efca3c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:05.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc8c0017f0 con 0x7efc9c13f710 2024-03-25T18:29:05.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca09fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:05.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca09fd640 1 -- 172.21.15.42:0/2711110009 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7efc8c001c00 con 0x7efc9c13f710 2024-03-25T18:29:05.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca3c63640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:05.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca3c63640 4 set_mon_vals no callback set 2024-03-25T18:29:05.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca09fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:05.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca09fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca09fd640 10 monclient: dump: 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:05.763 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:05.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca3c63640 10 monclient: shutdown 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca3c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.761+0000 7efca3c63640 1 -- 172.21.15.42:0/2711110009 >> v1:172.21.15.120:6789/0 conn(0x7efc9c13f710 legacy=0x7efc9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.762+0000 7efca3c63640 1 -- 172.21.15.42:0/2711110009 shutdown_connections 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.762+0000 7efca3c63640 1 -- 172.21.15.42:0/2711110009 >> 172.21.15.42:0/2711110009 conn(0x7efc9c0b7ea0 msgr2=0x7efc9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.762+0000 7efca3c63640 1 -- 172.21.15.42:0/2711110009 shutdown_connections 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.762+0000 7efca3c63640 1 -- 172.21.15.42:0/2711110009 wait complete. 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 10 monclient: build_initial_monmap 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 10 monclient: monmap: 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:05.766 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 1 librados: starting msgr at 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 1 librados: starting objecter 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 1 Processor -- start 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 1 -- start start 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 1 librados: setting wanted keys 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 1 librados: calling monclient init 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.763+0000 7efca3c63640 10 monclient: init 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:05.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 10 monclient(hunting): picked mon.noname-c con 0x7efc9c0a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:05.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 10 monclient(hunting): picked mon.noname-a con 0x7efc9c13f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:05.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 10 monclient(hunting): picked mon.noname-b con 0x7efc9c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:05.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c1dac50 con 0x7efc9c13f710 2024-03-25T18:29:05.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c0b16c0 con 0x7efc9c1430c0 2024-03-25T18:29:05.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c136b70 con 0x7efc9c0a98f0 2024-03-25T18:29:05.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:05.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.764+0000 7efca3c63640 10 monclient(hunting): authenticate will time out at 7860.991322s 2024-03-25T18:29:05.777 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.776+0000 7efca11fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc9c13f710 0x7efc9c1db0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46234/0 (socket says 172.21.15.42:46234) 2024-03-25T18:29:05.777 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.776+0000 7efca11fe640 1 -- 172.21.15.42:0/3794473210 learned_addr learned my addr 172.21.15.42:0/3794473210 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:05.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.776+0000 7efc927fc640 10 client.?.objecter ms_handle_connect 0x7efc9c1430c0 2024-03-25T18:29:05.778 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.776+0000 7efc927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:05.779 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.778+0000 7efc927fc640 10 client.?.objecter ms_handle_connect 0x7efc9c13f710 2024-03-25T18:29:05.779 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.778+0000 7efc927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:05.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.781+0000 7efc927fc640 10 client.?.objecter ms_handle_connect 0x7efc9c0a98f0 2024-03-25T18:29:05.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.781+0000 7efc927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:05.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.791+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338703387 0 0) 0x7efc9c0b16c0 con 0x7efc9c1430c0 2024-03-25T18:29:05.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.791+0000 7efc927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:05.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.791+0000 7efc927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:05.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.791+0000 7efc927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:05.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.791+0000 7efc927fc640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:29:05.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.791+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc80004eb0 con 0x7efc9c1430c0 2024-03-25T18:29:05.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.792+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874806419 0 0) 0x7efc9c1dac50 con 0x7efc9c13f710 2024-03-25T18:29:05.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.792+0000 7efc927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:05.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.792+0000 7efc927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:05.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.792+0000 7efc927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:05.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.792+0000 7efc927fc640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:29:05.794 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.792+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c0b16c0 con 0x7efc9c13f710 2024-03-25T18:29:05.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.797+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263276236 0 0) 0x7efc9c136b70 con 0x7efc9c0a98f0 2024-03-25T18:29:05.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.797+0000 7efc927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:05.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.797+0000 7efc927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:05.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.797+0000 7efc927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:05.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.797+0000 7efc927fc640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:29:05.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.797+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c1dac50 con 0x7efc9c0a98f0 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.808+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313854875 0 0) 0x7efc9c0b16c0 con 0x7efc9c13f710 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.808+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc9c136b70 con 0x7efc9c13f710 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.808+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7efc8c001b40 con 0x7efc9c13f710 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.808+0000 7efc927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.808+0000 7efc927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.808+0000 7efc927fc640 10 monclient(hunting): dump: 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:05.810 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:05.811 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:05.811 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:05.811 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:05.811 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.808+0000 7efc927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.813+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715295410 0 0) 0x7efc80004eb0 con 0x7efc9c1430c0 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.813+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc9c0b16c0 con 0x7efc9c1430c0 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.813+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7efc94001b90 con 0x7efc9c1430c0 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.813+0000 7efc927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.813+0000 7efc927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.813+0000 7efc927fc640 10 monclient(hunting): dump: 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:05.815 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:05.816 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:05.816 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:05.816 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:05.816 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2313926842 0 0) 0x7efc9c136b70 con 0x7efc9c13f710 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 >> v1:172.21.15.42:6790/0 conn(0x7efc9c0a98f0 legacy=0x7efc9c1375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 >> v1:172.21.15.120:6789/0 conn(0x7efc9c1430c0 legacy=0x7efc9c1e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 1 monclient: found mon.a 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:05.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c0b11a0 con 0x7efc9c13f710 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efc927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 10 monclient: _renew_subs 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c0b13f0 con 0x7efc9c13f710 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 10 monclient: _renew_subs 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c137ea0 con 0x7efc9c13f710 2024-03-25T18:29:05.821 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.818+0000 7efca3c63640 1 librados: init done 2024-03-25T18:29:05.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.820+0000 7efca3c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:05.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.820+0000 7efca3c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:05.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.820+0000 7efca3c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:05.822 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:05.820+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c0627b0 con 0x7efc9c13f710 2024-03-25T18:29:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.136+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc8c0011f0 con 0x7efc9c13f710 2024-03-25T18:29:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.136+0000 7efc927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.136+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7efc8c004c30 con 0x7efc9c13f710 2024-03-25T18:29:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.136+0000 7efc927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:06.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.136+0000 7efc927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.136+0000 7efc927fc640 10 monclient: dump: 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:06.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.136+0000 7efc93fff640 4 set_mon_vals no callback set 2024-03-25T18:29:06.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.137+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7efc8c01ac80 con 0x7efc9c13f710 2024-03-25T18:29:06.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.137+0000 7efca19ff640 10 monclient: get_auth_request con 0x7efc8006df20 auth_method 0 2024-03-25T18:29:06.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.137+0000 7efc927fc640 10 client.4653.objecter ms_handle_connect 0x7efc8006df20 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.143+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7efc8c054930 con 0x7efc9c13f710 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.143+0000 7efc927fc640 10 client.4653.objecter ms_dispatch 0x7efc9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.143+0000 7efc927fc640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.143+0000 7efc927fc640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.144+0000 7efc927fc640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.144+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc8c050270 con 0x7efc9c13f710 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.144+0000 7efc927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:06.145 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.144+0000 7efc927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:06.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.392+0000 7efca3c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:06.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.392+0000 7efca3c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:06.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.392+0000 7efca3c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:06.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.392+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc9c13e250 con 0x7efc9c13f710 2024-03-25T18:29:06.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.418+0000 7efc927fc640 1 -- 172.21.15.42:0/3794473210 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7efc8c052370 con 0x7efc9c13f710 2024-03-25T18:29:06.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.418+0000 7efc927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:06.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.418+0000 7efc927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:06.420 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:29:06.420 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:06.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 10 librados: watch_flush enter 2024-03-25T18:29:06.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 10 librados: watch_flush exit 2024-03-25T18:29:06.422 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 >> v1:172.21.15.42:6800/3081750888 conn(0x7efc8006df20 legacy=0x7efc800703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 10 monclient: shutdown 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 >> v1:172.21.15.42:6789/0 conn(0x7efc9c13f710 legacy=0x7efc9c1db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.420+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 shutdown_connections 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.421+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 >> 172.21.15.42:0/3794473210 conn(0x7efc9c0b7ea0 msgr2=0x7efc9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.421+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 shutdown_connections 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.421+0000 7efca3c63640 1 -- 172.21.15.42:0/3794473210 wait complete. 2024-03-25T18:29:06.423 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:06.421+0000 7efca3c63640 1 librados: shutdown 2024-03-25T18:29:06.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:13.242 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:13.269 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:56.438 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.325 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:56.495 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.338+0000 7f37304ed640 10 monclient: get_monmap_and_config 2024-03-25T18:29:13.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.338+0000 7f37304ed640 10 monclient: build_initial_monmap 2024-03-25T18:29:13.339 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.338+0000 7f37304ed640 10 monclient: monmap: 2024-03-25T18:29:13.339 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:13.339 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:13.339 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:13.339343+0000 2024-03-25T18:29:13.340 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:13.339343+0000 2024-03-25T18:29:13.340 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:13.340 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:13.340 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:13.340 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:13.340 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:13.340 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.338+0000 7f37304ed640 1 Processor -- start 2024-03-25T18:29:13.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.338+0000 7f37304ed640 1 -- start start 2024-03-25T18:29:13.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.338+0000 7f37304ed640 10 monclient: init 2024-03-25T18:29:13.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:13.341 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 10 monclient(hunting): picked mon.noname-a con 0x7f37280b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 10 monclient(hunting): picked mon.noname-c con 0x7f37280b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 10 monclient(hunting): picked mon.noname-b con 0x7f372814be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280af940 con 0x7f37280b4d60 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280b0a70 con 0x7f37280b1730 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280b0d50 con 0x7f372814be90 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:13.342 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.340+0000 7f37304ed640 10 monclient(hunting): authenticate will time out at 7868.566684s 2024-03-25T18:29:13.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.351+0000 7f372e289640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f37280b4d60 0x7f372813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54468/0 (socket says 172.21.15.42:54468) 2024-03-25T18:29:13.353 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.351+0000 7f372e289640 1 -- 172.21.15.42:0/4112471728 learned_addr learned my addr 172.21.15.42:0/4112471728 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:13.364 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:09:56.533 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.378 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:56.498 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.382 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:56.502 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.388+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180185658 0 0) 0x7f37280b0a70 con 0x7f37280b1730 2024-03-25T18:29:13.389 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.388+0000 7f372d287640 10 monclient(hunting): _negotiate 2024-03-25T18:29:13.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.388+0000 7f372d287640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:13.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.388+0000 7f372d287640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:13.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.388+0000 7f372d287640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:29:13.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.388+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714003520 con 0x7f37280b1730 2024-03-25T18:29:13.403 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:56.572 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.412 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:56.581 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.412+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850720794 0 0) 0x7f37280af940 con 0x7f37280b4d60 2024-03-25T18:29:13.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.412+0000 7f372d287640 10 monclient(hunting): _negotiate 2024-03-25T18:29:13.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.412+0000 7f372d287640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:13.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.412+0000 7f372d287640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:13.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.412+0000 7f372d287640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:29:13.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.412+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37280b0a70 con 0x7f37280b4d60 2024-03-25T18:29:13.415 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:09:56.535 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1058252727 0 0) 0x7f3714003520 con 0x7f37280b1730 2024-03-25T18:29:13.417 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 >> v1:172.21.15.42:6790/0 conn(0x7f372814be90 legacy=0x7f372814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 >> v1:172.21.15.42:6789/0 conn(0x7f37280b4d60 legacy=0x7f372813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 10 monclient: _finish_hunting 0 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 1 monclient: found mon.noname-c 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280b1030 con 0x7f37280b1730 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 10 monclient: _finish_auth 0 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 10 monclient: _check_auth_tickets 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3718001690 con 0x7f37280b1730 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f37304ed640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:29:13.418 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f37304ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f372d287640 10 monclient: dump: 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.419 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.416+0000 7f37304ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:13.443 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:09:56.612 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:13.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37180017f0 con 0x7f37280b1730 2024-03-25T18:29:13.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f372d287640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f372d287640 1 -- 172.21.15.42:0/4112471728 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3718001c00 con 0x7f37280b1730 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f372d287640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f372d287640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f372d287640 10 monclient: dump: 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:13.486 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:13.487 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:13.487 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:13.487 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f37304ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:13.487 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.484+0000 7f37304ed640 4 set_mon_vals no callback set 2024-03-25T18:29:13.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.485+0000 7f37304ed640 10 monclient: shutdown 2024-03-25T18:29:13.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.485+0000 7f37304ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:13.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.485+0000 7f37304ed640 1 -- 172.21.15.42:0/4112471728 >> v1:172.21.15.120:6789/0 conn(0x7f37280b1730 legacy=0x7f37280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:13.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.485+0000 7f37304ed640 1 -- 172.21.15.42:0/4112471728 shutdown_connections 2024-03-25T18:29:13.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.485+0000 7f37304ed640 1 -- 172.21.15.42:0/4112471728 >> 172.21.15.42:0/4112471728 conn(0x7f37280617a0 msgr2=0x7f3728061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.485+0000 7f37304ed640 1 -- 172.21.15.42:0/4112471728 shutdown_connections 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.485+0000 7f37304ed640 1 -- 172.21.15.42:0/4112471728 wait complete. 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 10 monclient: build_initial_monmap 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 10 monclient: monmap: 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:13.490 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 1 librados: starting msgr at 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 1 librados: starting objecter 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 1 Processor -- start 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 1 -- start start 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 1 librados: setting wanted keys 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 1 librados: calling monclient init 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.486+0000 7f37304ed640 10 monclient: init 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:13.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 10 monclient(hunting): picked mon.noname-c con 0x7f37280b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 10 monclient(hunting): picked mon.noname-b con 0x7f37280b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 10 monclient(hunting): picked mon.noname-a con 0x7f372814be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3728141d50 con 0x7f372814be90 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37281d8620 con 0x7f37280b4d60 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37281d79f0 con 0x7f37280b1730 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:13.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.487+0000 7f37304ed640 10 monclient(hunting): authenticate will time out at 7868.713729s 2024-03-25T18:29:13.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.511+0000 7f372da88640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f37280b4d60 0x7f37281405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:55116/0 (socket says 172.21.15.42:55116) 2024-03-25T18:29:13.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.511+0000 7f372da88640 1 -- 172.21.15.42:0/4265205726 learned_addr learned my addr 172.21.15.42:0/4265205726 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:13.513 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.512+0000 7f371effd640 10 client.?.objecter ms_handle_connect 0x7f37280b4d60 2024-03-25T18:29:13.514 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.512+0000 7f371effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:13.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.518+0000 7f371effd640 10 client.?.objecter ms_handle_connect 0x7f372814be90 2024-03-25T18:29:13.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.518+0000 7f371effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:13.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.532+0000 7f371effd640 10 client.?.objecter ms_handle_connect 0x7f37280b1730 2024-03-25T18:29:13.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.532+0000 7f371effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:13.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.540+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554741800 0 0) 0x7f37281d8620 con 0x7f37280b4d60 2024-03-25T18:29:13.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.540+0000 7f371effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:13.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.540+0000 7f371effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:13.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.540+0000 7f371effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:13.542 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.540+0000 7f371effd640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:29:13.543 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.540+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f370c004eb0 con 0x7f37280b4d60 2024-03-25T18:29:13.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.542+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255196639 0 0) 0x7f3728141d50 con 0x7f372814be90 2024-03-25T18:29:13.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.542+0000 7f371effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:13.544 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.542+0000 7f371effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:13.545 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.542+0000 7f371effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:13.545 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.542+0000 7f371effd640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:29:13.545 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.542+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37281d8620 con 0x7f372814be90 2024-03-25T18:29:13.552 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.550+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374196121 0 0) 0x7f37281d79f0 con 0x7f37280b1730 2024-03-25T18:29:13.552 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.550+0000 7f371effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:13.552 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.550+0000 7f371effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:13.552 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.550+0000 7f371effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:13.552 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.550+0000 7f371effd640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:29:13.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.550+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3728141d50 con 0x7f37280b1730 2024-03-25T18:29:13.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.562+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4059663683 0 0) 0x7f370c004eb0 con 0x7f37280b4d60 2024-03-25T18:29:13.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.562+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37281d79f0 con 0x7f37280b4d60 2024-03-25T18:29:13.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.562+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3718001bc0 con 0x7f37280b4d60 2024-03-25T18:29:13.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.562+0000 7f371effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:13.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.562+0000 7f371effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:13.565 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.562+0000 7f371effd640 10 monclient(hunting): dump: 2024-03-25T18:29:13.565 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:13.565 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:13.565 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.565 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.566 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:13.566 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:13.566 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:13.566 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:13.566 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:13.567 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.567 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.562+0000 7f371effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:13.569 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.567+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1730073588 0 0) 0x7f3728141d50 con 0x7f37280b1730 2024-03-25T18:29:13.569 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.567+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f370c004eb0 con 0x7f37280b1730 2024-03-25T18:29:13.569 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.567+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3724001930 con 0x7f37280b1730 2024-03-25T18:29:13.569 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.567+0000 7f371effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:13.569 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.567+0000 7f371effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:13.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.567+0000 7f371effd640 10 monclient(hunting): dump: 2024-03-25T18:29:13.570 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:13.570 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:13.570 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.570 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.570 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:13.571 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:13.571 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:13.571 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:13.571 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:13.571 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.577+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3167364804 0 0) 0x7f37281d8620 con 0x7f372814be90 2024-03-25T18:29:13.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.577+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3728141d50 con 0x7f372814be90 2024-03-25T18:29:13.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.577+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3720001b10 con 0x7f372814be90 2024-03-25T18:29:13.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.577+0000 7f371effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:13.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.577+0000 7f371effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:13.579 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.577+0000 7f371effd640 10 monclient(hunting): dump: 2024-03-25T18:29:13.580 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:13.580 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:13.580 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.580 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.580 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:13.580 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:13.581 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:13.581 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:13.581 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:13.581 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2865074217 0 0) 0x7f37281d79f0 con 0x7f37280b4d60 2024-03-25T18:29:13.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 >> v1:172.21.15.42:6790/0 conn(0x7f37280b1730 legacy=0x7f37281d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:13.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 >> v1:172.21.15.42:6789/0 conn(0x7f372814be90 legacy=0x7f3728142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:13.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:13.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 1 monclient: found mon.b 2024-03-25T18:29:13.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:13.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:13.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3728140d00 con 0x7f37280b4d60 2024-03-25T18:29:13.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 10 monclient: _finish_auth 0 2024-03-25T18:29:13.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:13.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f371effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:13.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:29:13.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 10 monclient: _renew_subs 2024-03-25T18:29:13.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:13.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372813b630 con 0x7f37280b4d60 2024-03-25T18:29:13.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:13.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 10 monclient: _renew_subs 2024-03-25T18:29:13.586 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:13.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3728143580 con 0x7f37280b4d60 2024-03-25T18:29:13.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.580+0000 7f37304ed640 1 librados: init done 2024-03-25T18:29:13.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.581+0000 7f37304ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:13.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.581+0000 7f37304ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:13.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.581+0000 7f37304ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:13.587 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.581+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37281d7830 con 0x7f37280b4d60 2024-03-25T18:29:13.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.864+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37180011f0 con 0x7f37280b4d60 2024-03-25T18:29:13.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.864+0000 7f371effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:13.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.864+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3718003b00 con 0x7f37280b4d60 2024-03-25T18:29:13.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.864+0000 7f371effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:13.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.864+0000 7f371effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:13.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.864+0000 7f371effd640 10 monclient: dump: 2024-03-25T18:29:13.868 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:13.868 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:13.868 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.868 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:13.869 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:13.869 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:13.869 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:13.869 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:13.869 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:13.869 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:13.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.864+0000 7f372ca86640 4 set_mon_vals no callback set 2024-03-25T18:29:13.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.865+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f3718003d20 con 0x7f37280b4d60 2024-03-25T18:29:13.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.866+0000 7f372e289640 10 monclient: get_auth_request con 0x7f370c06e0f0 auth_method 0 2024-03-25T18:29:13.870 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.866+0000 7f371effd640 10 client.4666.objecter ms_handle_connect 0x7f370c06e0f0 2024-03-25T18:29:13.872 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.871+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f3718054880 con 0x7f37280b4d60 2024-03-25T18:29:13.873 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.871+0000 7f371effd640 10 client.4666.objecter ms_dispatch 0x7f3728000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:13.873 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.871+0000 7f371effd640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:13.873 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.871+0000 7f371effd640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:13.873 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.871+0000 7f371effd640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T18:29:13.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.873+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3718050390 con 0x7f37280b4d60 2024-03-25T18:29:13.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.873+0000 7f371effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:13.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:13.873+0000 7f371effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:14.139 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.138+0000 7f37304ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:14.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.138+0000 7f37304ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:14.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.138+0000 7f37304ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:14.140 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.138+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37281d7b80 con 0x7f37280b4d60 2024-03-25T18:29:14.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.167+0000 7f371effd640 1 -- 172.21.15.42:0/4265205726 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f37180522c0 con 0x7f37280b4d60 2024-03-25T18:29:14.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.167+0000 7f371effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:14.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.167+0000 7f371effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:14.170 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:29:14.170 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:14.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 10 librados: watch_flush enter 2024-03-25T18:29:14.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 10 librados: watch_flush exit 2024-03-25T18:29:14.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T18:29:14.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 10 client.4666.objecter successfully canceled tick 2024-03-25T18:29:14.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 >> v1:172.21.15.42:6800/3081750888 conn(0x7f370c06e0f0 legacy=0x7f370c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:14.172 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 10 monclient: shutdown 2024-03-25T18:29:14.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:14.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 >> v1:172.21.15.120:6789/0 conn(0x7f37280b4d60 legacy=0x7f37281405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:14.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 shutdown_connections 2024-03-25T18:29:14.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 >> 172.21.15.42:0/4265205726 conn(0x7f37280617a0 msgr2=0x7f3728138590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:14.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 shutdown_connections 2024-03-25T18:29:14.173 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 1 -- 172.21.15.42:0/4265205726 wait complete. 2024-03-25T18:29:14.174 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:14.169+0000 7f37304ed640 1 librados: shutdown 2024-03-25T18:29:14.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:20.989 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:21.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.091+0000 7f1dca5fb640 10 monclient: get_monmap_and_config 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.091+0000 7f1dca5fb640 10 monclient: build_initial_monmap 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.091+0000 7f1dca5fb640 10 monclient: monmap: 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:21.092394+0000 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:21.092394+0000 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:21.093 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 1 Processor -- start 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 1 -- start start 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient: init 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc40b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc40b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc414be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:21.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc40af940 con 0x7f1dc414be90 2024-03-25T18:29:21.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc40b0a70 con 0x7f1dc40b1730 2024-03-25T18:29:21.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc40b0d50 con 0x7f1dc40b4d60 2024-03-25T18:29:21.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:21.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.092+0000 7f1dca5fb640 10 monclient(hunting): authenticate will time out at 7876.319309s 2024-03-25T18:29:21.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.108+0000 7f1dc37fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1dc40b1730 0x7f1dc40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:40384/0 (socket says 172.21.15.42:40384) 2024-03-25T18:29:21.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.108+0000 7f1dc37fe640 1 -- 172.21.15.42:0/3475684752 learned_addr learned my addr 172.21.15.42:0/3475684752 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:21.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.121+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855418110 0 0) 0x7f1dc40b0a70 con 0x7f1dc40b1730 2024-03-25T18:29:21.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.121+0000 7f1dc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:21.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.121+0000 7f1dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:21.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.121+0000 7f1dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:21.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.121+0000 7f1dc2ffd640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:29:21.123 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.121+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dac003520 con 0x7f1dc40b1730 2024-03-25T18:29:21.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.127+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688439454 0 0) 0x7f1dc40b0d50 con 0x7f1dc40b4d60 2024-03-25T18:29:21.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.127+0000 7f1dc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:21.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.127+0000 7f1dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:21.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.127+0000 7f1dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:21.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.127+0000 7f1dc2ffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:29:21.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.127+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc40b0a70 con 0x7f1dc40b4d60 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190879357 0 0) 0x7f1dac003520 con 0x7f1dc40b1730 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 >> v1:172.21.15.42:6790/0 conn(0x7f1dc40b4d60 legacy=0x7f1dc413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 >> v1:172.21.15.42:6789/0 conn(0x7f1dc414be90 legacy=0x7f1dc414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc40b1030 con 0x7f1dc40b1730 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:21.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1db8001690 con 0x7f1dc40b1730 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dc2ffd640 10 monclient: dump: 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dca5fb640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:29:21.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.133+0000 7f1dca5fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db80017f0 con 0x7f1dc40b1730 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dc2ffd640 1 -- 172.21.15.42:0/3475684752 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1db8001c00 con 0x7f1dc40b1730 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dc2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dc2ffd640 10 monclient: dump: 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:21.156 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dca5fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.150+0000 7f1dca5fb640 4 set_mon_vals no callback set 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.151+0000 7f1dca5fb640 10 monclient: shutdown 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.151+0000 7f1dca5fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.151+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3475684752 >> v1:172.21.15.120:6789/0 conn(0x7f1dc40b1730 legacy=0x7f1dc40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.151+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3475684752 shutdown_connections 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.151+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3475684752 >> 172.21.15.42:0/3475684752 conn(0x7f1dc40617a0 msgr2=0x7f1dc4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:21.157 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.151+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3475684752 shutdown_connections 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.151+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3475684752 wait complete. 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 10 monclient: build_initial_monmap 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 10 monclient: monmap: 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.158 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 1 librados: starting msgr at 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 1 librados: starting objecter 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 1 Processor -- start 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 1 -- start start 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 1 librados: setting wanted keys 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 1 librados: calling monclient init 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.152+0000 7f1dca5fb640 10 monclient: init 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc40b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc40b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc414be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc413aa70 con 0x7f1dc414be90 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc413ad50 con 0x7f1dc40b1730 2024-03-25T18:29:21.159 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc4138440 con 0x7f1dc40b4d60 2024-03-25T18:29:21.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:21.160 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.153+0000 7f1dca5fb640 10 monclient(hunting): authenticate will time out at 7876.380311s 2024-03-25T18:29:21.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.162+0000 7f1dc3fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f1dc40b1730 0x7f1dc4139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:40434/0 (socket says 172.21.15.42:40434) 2024-03-25T18:29:21.163 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.162+0000 7f1dc3fff640 1 -- 172.21.15.42:0/3996258491 learned_addr learned my addr 172.21.15.42:0/3996258491 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:21.164 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.163+0000 7f1dc0ff9640 10 client.?.objecter ms_handle_connect 0x7f1dc40b1730 2024-03-25T18:29:21.165 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.163+0000 7f1dc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:21.166 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.165+0000 7f1dc0ff9640 10 client.?.objecter ms_handle_connect 0x7f1dc414be90 2024-03-25T18:29:21.166 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.165+0000 7f1dc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:21.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.166+0000 7f1dc0ff9640 10 client.?.objecter ms_handle_connect 0x7f1dc40b4d60 2024-03-25T18:29:21.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.166+0000 7f1dc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:21.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707406084 0 0) 0x7f1dc413ad50 con 0x7f1dc40b1730 2024-03-25T18:29:21.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:21.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0004eb0 con 0x7f1dc40b1730 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028778544 0 0) 0x7f1dc413aa70 con 0x7f1dc414be90 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:29:21.178 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.176+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc413ad50 con 0x7f1dc414be90 2024-03-25T18:29:21.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.187+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788023789 0 0) 0x7f1dc4138440 con 0x7f1dc40b4d60 2024-03-25T18:29:21.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.187+0000 7f1dc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:21.189 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.187+0000 7f1dc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:21.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.187+0000 7f1dc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:21.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.187+0000 7f1dc0ff9640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:29:21.190 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.187+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc413aa70 con 0x7f1dc40b4d60 2024-03-25T18:29:21.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.191+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625537338 0 0) 0x7f1dc413ad50 con 0x7f1dc414be90 2024-03-25T18:29:21.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.191+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc4138440 con 0x7f1dc414be90 2024-03-25T18:29:21.192 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.191+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1dbc001a70 con 0x7f1dc414be90 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.191+0000 7f1dc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.191+0000 7f1dc0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.191+0000 7f1dc0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:21.193 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:21.194 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.191+0000 7f1dc0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:21.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.193+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838150126 0 0) 0x7f1db0004eb0 con 0x7f1dc40b1730 2024-03-25T18:29:21.194 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.193+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc413ad50 con 0x7f1dc40b1730 2024-03-25T18:29:21.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.198+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1045139076 0 0) 0x7f1dc413aa70 con 0x7f1dc40b4d60 2024-03-25T18:29:21.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.198+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1db0004eb0 con 0x7f1dc40b4d60 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.200+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1db4001ab0 con 0x7f1dc40b1730 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.200+0000 7f1dc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.200+0000 7f1dc0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.200+0000 7f1dc0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:21.201 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:21.202 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:21.202 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:21.202 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:21.202 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.207 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.205+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1db8001b60 con 0x7f1dc40b4d60 2024-03-25T18:29:21.207 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.205+0000 7f1dc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:21.207 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.205+0000 7f1dc0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:21.207 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.205+0000 7f1dc0ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:21.207 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:21.207 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:21.207 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.208 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.208 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:21.208 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:21.208 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:21.208 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:21.208 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:21.208 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 501553083 0 0) 0x7f1dc4138440 con 0x7f1dc414be90 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 >> v1:172.21.15.42:6790/0 conn(0x7f1dc40b4d60 legacy=0x7f1dc41310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 >> v1:172.21.15.120:6789/0 conn(0x7f1dc40b1730 legacy=0x7f1dc4139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 1 monclient: found mon.a 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc4138720 con 0x7f1dc414be90 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:29:21.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 10 monclient: _renew_subs 2024-03-25T18:29:21.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:21.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc4139770 con 0x7f1dc414be90 2024-03-25T18:29:21.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 10 client.4662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:21.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 10 monclient: _renew_subs 2024-03-25T18:29:21.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:21.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc41317b0 con 0x7f1dc414be90 2024-03-25T18:29:21.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.209+0000 7f1dca5fb640 1 librados: init done 2024-03-25T18:29:21.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.210+0000 7f1dca5fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:21.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.210+0000 7f1dca5fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:21.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.210+0000 7f1dca5fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:21.213 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.210+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc413fd90 con 0x7f1dc414be90 2024-03-25T18:29:21.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.218+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dbc001f60 con 0x7f1dc414be90 2024-03-25T18:29:21.219 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.218+0000 7f1dc0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:21.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.218+0000 7f1dc27fc640 4 set_mon_vals no callback set 2024-03-25T18:29:21.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.529+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f1dbc003b10 con 0x7f1dc414be90 2024-03-25T18:29:21.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.529+0000 7f1dc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:21.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.529+0000 7f1dc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:21.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.529+0000 7f1dc0ff9640 10 monclient: dump: 2024-03-25T18:29:21.531 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:21.531 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:21.531 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.532 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:21.532 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:21.532 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:21.532 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:21.532 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:21.532 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:21.532 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:21.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.530+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f1dbc019ae0 con 0x7f1dc414be90 2024-03-25T18:29:21.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.531+0000 7f1dc3fff640 10 monclient: get_auth_request con 0x7f1db006e040 auth_method 0 2024-03-25T18:29:21.533 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.531+0000 7f1dc0ff9640 10 client.4662.objecter ms_handle_connect 0x7f1db006e040 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.538+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f1dbc0536b0 con 0x7f1dc414be90 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.538+0000 7f1dc0ff9640 10 client.4662.objecter ms_dispatch 0x7f1dc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.538+0000 7f1dc0ff9640 3 client.4662.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.538+0000 7f1dc0ff9640 3 client.4662.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.539+0000 7f1dc0ff9640 20 client.4662.objecter dump_active .. 0 homeless 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.539+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dbc04eff0 con 0x7f1dc414be90 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.539+0000 7f1dc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:21.540 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.539+0000 7f1dc0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:21.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.805+0000 7f1dca5fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:21.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.805+0000 7f1dca5fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:21.806 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.805+0000 7f1dca5fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:29:21.807 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.805+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dc41388b0 con 0x7f1dc414be90 2024-03-25T18:29:21.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.838+0000 7f1dc0ff9640 1 -- 172.21.15.42:0/3996258491 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f1dbc0510f0 con 0x7f1dc414be90 2024-03-25T18:29:21.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.839+0000 7f1dc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:21.840 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.839+0000 7f1dc0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:21.841 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:29:21.841 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:21.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.840+0000 7f1dca5fb640 10 librados: watch_flush enter 2024-03-25T18:29:21.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 10 librados: watch_flush exit 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 20 client.4662.objecter shutdown clearing up homeless session... 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 10 client.4662.objecter successfully canceled tick 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 >> v1:172.21.15.42:6800/3081750888 conn(0x7f1db006e040 legacy=0x7f1db0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 10 monclient: shutdown 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 >> v1:172.21.15.42:6789/0 conn(0x7f1dc414be90 legacy=0x7f1dc413b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 shutdown_connections 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 >> 172.21.15.42:0/3996258491 conn(0x7f1dc40617a0 msgr2=0x7f1dc413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 shutdown_connections 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 1 -- 172.21.15.42:0/3996258491 wait complete. 2024-03-25T18:29:21.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:21.841+0000 7f1dca5fb640 1 librados: shutdown 2024-03-25T18:29:21.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:28.660 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.764+0000 7fb7fce8f640 10 monclient: get_monmap_and_config 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.764+0000 7fb7fce8f640 10 monclient: build_initial_monmap 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.764+0000 7fb7fce8f640 10 monclient: monmap: 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:28.765186+0000 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:28.765186+0000 2024-03-25T18:29:28.765 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:28.766 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:28.766 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:28.766 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:28.766 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:28.766 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:28.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.764+0000 7fb7fce8f640 1 Processor -- start 2024-03-25T18:29:28.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.764+0000 7fb7fce8f640 1 -- start start 2024-03-25T18:29:28.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.764+0000 7fb7fce8f640 10 monclient: init 2024-03-25T18:29:28.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:28.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:28.767 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f80b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:28.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7f80b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:28.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f814be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:28.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f80af940 con 0x7fb7f80b4d60 2024-03-25T18:29:28.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f80b0a70 con 0x7fb7f80b1730 2024-03-25T18:29:28.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f80b0d50 con 0x7fb7f814be90 2024-03-25T18:29:28.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:28.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.765+0000 7fb7fce8f640 10 monclient(hunting): authenticate will time out at 7883.992114s 2024-03-25T18:29:28.802 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.800+0000 7fb7f5d9b640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fb7f80b1730 0x7fb7f80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:56218/0 (socket says 172.21.15.42:56218) 2024-03-25T18:29:28.802 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.800+0000 7fb7f5d9b640 1 -- 172.21.15.42:0/2017098199 learned_addr learned my addr 172.21.15.42:0/2017098199 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:28.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.829+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142280424 0 0) 0x7fb7f80b0a70 con 0x7fb7f80b1730 2024-03-25T18:29:28.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.829+0000 7fb7f559a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:28.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.829+0000 7fb7f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:28.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.829+0000 7fb7f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:28.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.829+0000 7fb7f559a640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:29:28.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.829+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e0003520 con 0x7fb7f80b1730 2024-03-25T18:29:28.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437689434 0 0) 0x7fb7f80af940 con 0x7fb7f80b4d60 2024-03-25T18:29:28.842 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f80b0a70 con 0x7fb7f80b4d60 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3891756765 0 0) 0x7fb7e0003520 con 0x7fb7f80b1730 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 >> v1:172.21.15.42:6790/0 conn(0x7fb7f814be90 legacy=0x7fb7f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 >> v1:172.21.15.42:6789/0 conn(0x7fb7f80b4d60 legacy=0x7fb7f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 monclient: found mon.noname-c 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80b1030 con 0x7fb7f80b1730 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient: _finish_auth 0 2024-03-25T18:29:28.843 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb7ec001690 con 0x7fb7f80b1730 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7f559a640 10 monclient: dump: 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:28.844 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7fce8f640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:29:28.845 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.841+0000 7fb7fce8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.845+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7ec0017f0 con 0x7fb7f80b1730 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.845+0000 7fb7f559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.845+0000 7fb7f559a640 1 -- 172.21.15.42:0/2017098199 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb7ec001c00 con 0x7fb7f80b1730 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.845+0000 7fb7f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.846+0000 7fb7f559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.846+0000 7fb7f559a640 10 monclient: dump: 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:28.847 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.846+0000 7fb7fce8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:28.848 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.846+0000 7fb7fce8f640 4 set_mon_vals no callback set 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.846+0000 7fb7fce8f640 10 monclient: shutdown 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.846+0000 7fb7fce8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.846+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2017098199 >> v1:172.21.15.120:6789/0 conn(0x7fb7f80b1730 legacy=0x7fb7f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.847+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2017098199 shutdown_connections 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.847+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2017098199 >> 172.21.15.42:0/2017098199 conn(0x7fb7f80617a0 msgr2=0x7fb7f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.847+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2017098199 shutdown_connections 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.847+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2017098199 wait complete. 2024-03-25T18:29:28.850 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.847+0000 7fb7fce8f640 10 monclient: build_initial_monmap 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 10 monclient: monmap: 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 1 librados: starting msgr at 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 1 librados: starting objecter 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 1 Processor -- start 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 1 -- start start 2024-03-25T18:29:28.851 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 1 librados: setting wanted keys 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 1 librados: calling monclient init 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 10 monclient: init 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.848+0000 7fb7fce8f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f80b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.849+0000 7fb7fce8f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7f80b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.849+0000 7fb7fce8f640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f814be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.849+0000 7fb7fce8f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f812f7b0 con 0x7fb7f814be90 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.849+0000 7fb7fce8f640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f813adf0 con 0x7fb7f80b1730 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.849+0000 7fb7fce8f640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f813b0d0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.849+0000 7fb7fce8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:28.852 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.849+0000 7fb7fce8f640 10 monclient(hunting): authenticate will time out at 7884.075660s 2024-03-25T18:29:28.853 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.851+0000 7fb7f5d9b640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7fb7f80b4d60 0x7fb7f812e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:58336/0 (socket says 172.21.15.42:58336) 2024-03-25T18:29:28.853 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.851+0000 7fb7f5d9b640 1 -- 172.21.15.42:0/2716951211 learned_addr learned my addr 172.21.15.42:0/2716951211 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:28.853 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.852+0000 7fb7e6ffd640 10 client.?.objecter ms_handle_connect 0x7fb7f80b4d60 2024-03-25T18:29:28.853 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.852+0000 7fb7e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:28.854 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.853+0000 7fb7e6ffd640 10 client.?.objecter ms_handle_connect 0x7fb7f80b1730 2024-03-25T18:29:28.854 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.853+0000 7fb7e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:28.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.864+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389362279 0 0) 0x7fb7f813b0d0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.864+0000 7fb7e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:28.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.864+0000 7fb7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:28.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.864+0000 7fb7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:28.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.864+0000 7fb7e6ffd640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:29:28.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.864+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d8004eb0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.867 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.866+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945962188 0 0) 0x7fb7f813adf0 con 0x7fb7f80b1730 2024-03-25T18:29:28.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.866+0000 7fb7e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:28.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.866+0000 7fb7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:28.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.866+0000 7fb7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:28.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.866+0000 7fb7e6ffd640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:29:28.868 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.866+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f813b0d0 con 0x7fb7f80b1730 2024-03-25T18:29:28.872 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.871+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1796114452 0 0) 0x7fb7d8004eb0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.872 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.871+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7f813adf0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb7ec0047e0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:28.887 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2481323003 0 0) 0x7fb7f813adf0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 >> v1:172.21.15.120:6789/0 conn(0x7fb7f80b1730 legacy=0x7fb7f8139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 >> v1:172.21.15.42:6789/0 conn(0x7fb7f814be90 legacy=0x7fb7f812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 1 monclient: found mon.c 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f813b3b0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 5 monclient: authenticate success, global_id 4673 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 10 monclient: _renew_subs 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:29:28.888 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f813bdd0 con 0x7fb7f80b4d60 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 10 monclient: _renew_subs 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f8145690 con 0x7fb7f80b4d60 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.884+0000 7fb7fce8f640 1 librados: init done 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.885+0000 7fb7fce8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.885+0000 7fb7fce8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.885+0000 7fb7fce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:29:28.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:28.885+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f81401f0 con 0x7fb7f80b4d60 2024-03-25T18:29:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.196+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7ec004bf0 con 0x7fb7f80b4d60 2024-03-25T18:29:29.197 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.196+0000 7fb7e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.196+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fb7ec004f00 con 0x7fb7f80b4d60 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.196+0000 7fb7e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.196+0000 7fb7e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.196+0000 7fb7e6ffd640 10 monclient: dump: 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:29.198 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:29.199 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:29.199 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:29.199 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.196+0000 7fb7f4d99640 4 set_mon_vals no callback set 2024-03-25T18:29:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.197+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fb7ec01aed0 con 0x7fb7f80b4d60 2024-03-25T18:29:29.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.197+0000 7fb7f659c640 10 monclient: get_auth_request con 0x7fb7d806dce0 auth_method 0 2024-03-25T18:29:29.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.197+0000 7fb7e6ffd640 10 client.4673.objecter ms_handle_connect 0x7fb7d806dce0 2024-03-25T18:29:29.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7fb7ec0549e0 con 0x7fb7f80b4d60 2024-03-25T18:29:29.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 10 client.4673.objecter ms_dispatch 0x7fb7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:29.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 3 client.4673.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:29.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 3 client.4673.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:29.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-25T18:29:29.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7ec0504f0 con 0x7fb7f80b4d60 2024-03-25T18:29:29.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:29.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.209+0000 7fb7e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:29.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.462+0000 7fb7fce8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:29.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.462+0000 7fb7fce8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:29.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.462+0000 7fb7fce8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:29:29.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.462+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7f8145930 con 0x7fb7f80b4d60 2024-03-25T18:29:29.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.490+0000 7fb7e6ffd640 1 -- 172.21.15.42:0/2716951211 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7fb7ec052420 con 0x7fb7f80b4d60 2024-03-25T18:29:29.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.490+0000 7fb7e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:29.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.490+0000 7fb7e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:29.491 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:29:29.492 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 10 librados: watch_flush enter 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 10 librados: watch_flush exit 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 10 client.4673.objecter successfully canceled tick 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 >> v1:172.21.15.42:6800/3081750888 conn(0x7fb7d806dce0 legacy=0x7fb7d80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 10 monclient: shutdown 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 >> v1:172.21.15.42:6790/0 conn(0x7fb7f80b4d60 legacy=0x7fb7f812e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:29.494 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 shutdown_connections 2024-03-25T18:29:29.495 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 >> 172.21.15.42:0/2716951211 conn(0x7fb7f80617a0 msgr2=0x7fb7f813c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:29.495 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 shutdown_connections 2024-03-25T18:29:29.495 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 1 -- 172.21.15.42:0/2716951211 wait complete. 2024-03-25T18:29:29.495 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:29.492+0000 7fb7fce8f640 1 librados: shutdown 2024-03-25T18:29:29.506 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:33.358 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:16.527 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.403 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:16.524 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.409 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:16.578 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.418 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:16.538 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.446 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:16.615 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.450 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:16.570 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.473 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:16.642 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.477 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:16.647 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:33.512 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:16.682 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:36.313 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:36.425 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.424+0000 7f8a8dc30640 10 monclient: get_monmap_and_config 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.424+0000 7f8a8dc30640 10 monclient: build_initial_monmap 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.424+0000 7f8a8dc30640 10 monclient: monmap: 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:36.425632+0000 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:36.425632+0000 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:36.426 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:36.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.425+0000 7f8a8dc30640 1 Processor -- start 2024-03-25T18:29:36.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.425+0000 7f8a8dc30640 1 -- start start 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.425+0000 7f8a8dc30640 10 monclient: init 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 10 monclient(hunting): picked mon.noname-a con 0x7f8a880b4f40 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 10 monclient(hunting): picked mon.noname-b con 0x7f8a8813fd90 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 10 monclient(hunting): picked mon.noname-c con 0x7f8a880b4540 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a880aa0c0 con 0x7f8a880b4f40 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a880ab170 con 0x7f8a880b4540 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a880a9ba0 con 0x7f8a8813fd90 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:36.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.426+0000 7f8a8dc30640 10 monclient(hunting): authenticate will time out at 7891.652869s 2024-03-25T18:29:36.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.435+0000 7f8a86ffd640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f8a8813fd90 0x7f8a881423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:34624/0 (socket says 172.21.15.42:34624) 2024-03-25T18:29:36.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.435+0000 7f8a86ffd640 1 -- 172.21.15.42:0/1349064389 learned_addr learned my addr 172.21.15.42:0/1349064389 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:36.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.449+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641782864 0 0) 0x7f8a880ab170 con 0x7f8a880b4540 2024-03-25T18:29:36.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.449+0000 7f8a867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:36.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.449+0000 7f8a867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:36.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.449+0000 7f8a867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:36.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.449+0000 7f8a867fc640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:29:36.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.449+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a70003520 con 0x7f8a880b4540 2024-03-25T18:29:36.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.451+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581992806 0 0) 0x7f8a880a9ba0 con 0x7f8a8813fd90 2024-03-25T18:29:36.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.451+0000 7f8a867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:36.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.451+0000 7f8a867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:36.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.452+0000 7f8a867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:36.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.452+0000 7f8a867fc640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:29:36.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.452+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a880ab170 con 0x7f8a8813fd90 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 56914388 0 0) 0x7f8a70003520 con 0x7f8a880b4540 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 >> v1:172.21.15.42:6790/0 conn(0x7f8a8813fd90 legacy=0x7f8a881423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 >> v1:172.21.15.42:6789/0 conn(0x7f8a880b4f40 legacy=0x7f8a8813f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a880a9680 con 0x7f8a880b4540 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:36.459 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8a80001a10 con 0x7f8a880b4540 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a8dc30640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a8dc30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a867fc640 10 monclient: dump: 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:36.460 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:36.461 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.457+0000 7f8a8dc30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a80001b70 con 0x7f8a880b4540 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a867fc640 1 -- 172.21.15.42:0/1349064389 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8a80001f80 con 0x7f8a880b4540 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a867fc640 10 monclient: dump: 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:36.464 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:36.465 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:36.465 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:36.465 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:36.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a8dc30640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:36.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.462+0000 7f8a8dc30640 4 set_mon_vals no callback set 2024-03-25T18:29:36.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.463+0000 7f8a8dc30640 10 monclient: shutdown 2024-03-25T18:29:36.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.463+0000 7f8a8dc30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:36.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.463+0000 7f8a8dc30640 1 -- 172.21.15.42:0/1349064389 >> v1:172.21.15.120:6789/0 conn(0x7f8a880b4540 legacy=0x7f8a8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:36.466 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.463+0000 7f8a8dc30640 1 -- 172.21.15.42:0/1349064389 shutdown_connections 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.463+0000 7f8a8dc30640 1 -- 172.21.15.42:0/1349064389 >> 172.21.15.42:0/1349064389 conn(0x7f8a880617a0 msgr2=0x7f8a88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.463+0000 7f8a8dc30640 1 -- 172.21.15.42:0/1349064389 shutdown_connections 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.463+0000 7f8a8dc30640 1 -- 172.21.15.42:0/1349064389 wait complete. 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.464+0000 7f8a8dc30640 10 monclient: build_initial_monmap 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.464+0000 7f8a8dc30640 10 monclient: monmap: 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:36.467 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:36.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.465+0000 7f8a8dc30640 1 librados: starting msgr at 2024-03-25T18:29:36.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.465+0000 7f8a8dc30640 1 librados: starting objecter 2024-03-25T18:29:36.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.465+0000 7f8a8dc30640 1 Processor -- start 2024-03-25T18:29:36.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.465+0000 7f8a8dc30640 1 -- start start 2024-03-25T18:29:36.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.465+0000 7f8a8dc30640 1 librados: setting wanted keys 2024-03-25T18:29:36.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.465+0000 7f8a8dc30640 1 librados: calling monclient init 2024-03-25T18:29:36.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.465+0000 7f8a8dc30640 10 monclient: init 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 10 monclient(hunting): picked mon.noname-b con 0x7f8a880b4540 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 10 monclient(hunting): picked mon.noname-c con 0x7f8a880b4f40 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 10 monclient(hunting): picked mon.noname-a con 0x7f8a8813fd90 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a881db6c0 con 0x7f8a8813fd90 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a881db1a0 con 0x7f8a880b4540 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a881db330 con 0x7f8a880b4f40 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:36.469 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.466+0000 7f8a8dc30640 10 monclient(hunting): authenticate will time out at 7891.692709s 2024-03-25T18:29:36.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.481+0000 7f8a877fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8a880b4540 0x7f8a881daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:60848/0 (socket says 172.21.15.42:60848) 2024-03-25T18:29:36.482 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.481+0000 7f8a877fe640 1 -- 172.21.15.42:0/3723127042 learned_addr learned my addr 172.21.15.42:0/3723127042 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:36.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.484+0000 7f8a6ffff640 10 client.?.objecter ms_handle_connect 0x7f8a880b4540 2024-03-25T18:29:36.485 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.484+0000 7f8a6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:36.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.488+0000 7f8a6ffff640 10 client.?.objecter ms_handle_connect 0x7f8a880b4f40 2024-03-25T18:29:36.489 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.488+0000 7f8a6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:36.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.489+0000 7f8a6ffff640 10 client.?.objecter ms_handle_connect 0x7f8a8813fd90 2024-03-25T18:29:36.490 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.489+0000 7f8a6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:36.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.492+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925168571 0 0) 0x7f8a881db1a0 con 0x7f8a880b4540 2024-03-25T18:29:36.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.492+0000 7f8a6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:36.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.492+0000 7f8a6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:36.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.492+0000 7f8a6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:36.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.492+0000 7f8a6ffff640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:29:36.493 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.492+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a74004eb0 con 0x7f8a880b4540 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.500+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2444697212 0 0) 0x7f8a74004eb0 con 0x7f8a880b4540 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.500+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a881db1a0 con 0x7f8a880b4540 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.500+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8a78001a30 con 0x7f8a880b4540 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.500+0000 7f8a6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.500+0000 7f8a6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.500+0000 7f8a6ffff640 10 monclient(hunting): dump: 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.502 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:36.503 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:36.503 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:36.503 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:36.503 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:36.503 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:36.503 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.500+0000 7f8a6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:36.505 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.504+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928992810 0 0) 0x7f8a881db330 con 0x7f8a880b4f40 2024-03-25T18:29:36.505 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.504+0000 7f8a6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:36.506 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.504+0000 7f8a6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:36.506 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.504+0000 7f8a6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:36.506 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.504+0000 7f8a6ffff640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:29:36.506 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.504+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a74004eb0 con 0x7f8a880b4f40 2024-03-25T18:29:36.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3612106354 0 0) 0x7f8a881db1a0 con 0x7f8a880b4540 2024-03-25T18:29:36.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 >> v1:172.21.15.42:6790/0 conn(0x7f8a880b4f40 legacy=0x7f8a881d45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:36.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 >> v1:172.21.15.42:6789/0 conn(0x7f8a8813fd90 legacy=0x7f8a880ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:36.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 1 monclient: found mon.b 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a880b1050 con 0x7f8a880b4540 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 10 monclient: _finish_auth 0 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 10 monclient: _renew_subs 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a880b12a0 con 0x7f8a880b4540 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 10 monclient: _renew_subs 2024-03-25T18:29:36.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:36.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a881d4eb0 con 0x7f8a880b4540 2024-03-25T18:29:36.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.506+0000 7f8a8dc30640 1 librados: init done 2024-03-25T18:29:36.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.507+0000 7f8a8dc30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:36.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.507+0000 7f8a8dc30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:36.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.507+0000 7f8a8dc30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:36.509 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.507+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a881d5160 con 0x7f8a880b4540 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.773+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a78001cb0 con 0x7f8a880b4540 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.773+0000 7f8a6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.773+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8a78003a60 con 0x7f8a880b4540 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.773+0000 7f8a6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.773+0000 7f8a6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.773+0000 7f8a6ffff640 10 monclient: dump: 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:36.775 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:36.776 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:36.776 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:36.776 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:36.776 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:36.776 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.773+0000 7f8a85ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:36.776 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.774+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f8a78003c40 con 0x7f8a880b4540 2024-03-25T18:29:36.776 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.775+0000 7f8a86ffd640 10 monclient: get_auth_request con 0x7f8a7406dd60 auth_method 0 2024-03-25T18:29:36.777 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.775+0000 7f8a6ffff640 10 client.4684.objecter ms_handle_connect 0x7f8a7406dd60 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.780+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 128560285 0 0) 0x7f8a78054960 con 0x7f8a880b4540 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.780+0000 7f8a6ffff640 10 client.4684.objecter ms_dispatch 0x7f8a88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.780+0000 7f8a6ffff640 3 client.4684.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.780+0000 7f8a6ffff640 3 client.4684.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.780+0000 7f8a6ffff640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.781+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a780502a0 con 0x7f8a880b4540 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.781+0000 7f8a6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:36.782 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:36.781+0000 7f8a6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:37.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.026+0000 7f8a8dc30640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:37.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.026+0000 7f8a8dc30640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:37.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.026+0000 7f8a8dc30640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:37.027 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.026+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a880b09e0 con 0x7f8a880b4540 2024-03-25T18:29:37.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.048+0000 7f8a6ffff640 1 -- 172.21.15.42:0/3723127042 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1476121534) 0x7f8a780523a0 con 0x7f8a880b4540 2024-03-25T18:29:37.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.048+0000 7f8a6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:37.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.048+0000 7f8a6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:37.050 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:29:37.050 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:19:44.776741+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:44.776741+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 10 librados: watch_flush enter 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 10 librados: watch_flush exit 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 10 client.4684.objecter successfully canceled tick 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 >> v1:172.21.15.42:6800/3081750888 conn(0x7f8a7406dd60 legacy=0x7f8a74070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 10 monclient: shutdown 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 >> v1:172.21.15.120:6789/0 conn(0x7f8a880b4540 legacy=0x7f8a881daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 shutdown_connections 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 >> 172.21.15.42:0/3723127042 conn(0x7f8a880617a0 msgr2=0x7f8a88139d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:37.053 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.051+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 shutdown_connections 2024-03-25T18:29:37.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.052+0000 7f8a8dc30640 1 -- 172.21.15.42:0/3723127042 wait complete. 2024-03-25T18:29:37.054 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:37.052+0000 7f8a8dc30640 1 librados: shutdown 2024-03-25T18:29:37.064 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:43.872 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.982+0000 7f0bb8886640 10 monclient: get_monmap_and_config 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.982+0000 7f0bb8886640 10 monclient: build_initial_monmap 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.982+0000 7f0bb8886640 10 monclient: monmap: 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:43.983776+0000 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:43.983776+0000 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:43.984 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:43.985 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.983+0000 7f0bb8886640 1 Processor -- start 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.983+0000 7f0bb8886640 1 -- start start 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.983+0000 7f0bb8886640 10 monclient: init 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 10 monclient(hunting): picked mon.noname-b con 0x7f0bb406c380 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 10 monclient(hunting): picked mon.noname-a con 0x7f0bb406d670 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 10 monclient(hunting): picked mon.noname-c con 0x7f0bb40af8f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb406c9a0 con 0x7f0bb406d670 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb40ada90 con 0x7f0bb40af8f0 2024-03-25T18:29:43.986 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb40b0d10 con 0x7f0bb406c380 2024-03-25T18:29:43.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:43.987 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.984+0000 7f0bb8886640 10 monclient(hunting): authenticate will time out at 7899.211227s 2024-03-25T18:29:43.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.995+0000 7f0bb1d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0bb406d670 0x7f0bb40ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56418/0 (socket says 172.21.15.42:56418) 2024-03-25T18:29:43.996 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:43.995+0000 7f0bb1d9b640 1 -- 172.21.15.42:0/3929812519 learned_addr learned my addr 172.21.15.42:0/3929812519 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:44.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.010+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546522005 0 0) 0x7f0bb406c9a0 con 0x7f0bb406d670 2024-03-25T18:29:44.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.010+0000 7f0bb159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:44.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.010+0000 7f0bb159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:44.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.010+0000 7f0bb159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:44.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.010+0000 7f0bb159a640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:29:44.012 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.010+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b9c003520 con 0x7f0bb406d670 2024-03-25T18:29:44.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.020+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511485016 0 0) 0x7f0bb40ada90 con 0x7f0bb40af8f0 2024-03-25T18:29:44.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.020+0000 7f0bb159a640 10 monclient(hunting): _negotiate 2024-03-25T18:29:44.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.020+0000 7f0bb159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:44.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.020+0000 7f0bb159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:44.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.020+0000 7f0bb159a640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:29:44.022 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.021+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb406c9a0 con 0x7f0bb40af8f0 2024-03-25T18:29:44.023 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719827620 0 0) 0x7f0b9c003520 con 0x7f0bb406d670 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 >> v1:172.21.15.42:6790/0 conn(0x7f0bb406c380 legacy=0x7f0bb40b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 >> v1:172.21.15.120:6789/0 conn(0x7f0bb40af8f0 legacy=0x7f0bb40aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 10 monclient: _finish_hunting 0 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 1 monclient: found mon.noname-a 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb40add00 con 0x7f0bb406d670 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 10 monclient: _finish_auth 0 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 10 monclient: _check_auth_tickets 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0ba8001690 con 0x7f0bb406d670 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:44.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb159a640 10 monclient: dump: 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb8886640 5 monclient: authenticate success, global_id 4674 2024-03-25T18:29:44.025 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.022+0000 7f0bb8886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:44.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba80017f0 con 0x7f0bb406d670 2024-03-25T18:29:44.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:44.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb159a640 1 -- 172.21.15.42:0/3929812519 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0ba8001b80 con 0x7f0bb406d670 2024-03-25T18:29:44.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb8886640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:44.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb8886640 4 set_mon_vals no callback set 2024-03-25T18:29:44.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb159a640 10 monclient: dump: 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:44.045 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:44.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb8886640 10 monclient: shutdown 2024-03-25T18:29:44.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb8886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:44.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.043+0000 7f0bb8886640 1 -- 172.21.15.42:0/3929812519 >> v1:172.21.15.42:6789/0 conn(0x7f0bb406d670 legacy=0x7f0bb40ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:44.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.044+0000 7f0bb8886640 1 -- 172.21.15.42:0/3929812519 shutdown_connections 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.044+0000 7f0bb8886640 1 -- 172.21.15.42:0/3929812519 >> 172.21.15.42:0/3929812519 conn(0x7f0bb40b5c00 msgr2=0x7f0bb40b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.044+0000 7f0bb8886640 1 -- 172.21.15.42:0/3929812519 shutdown_connections 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.044+0000 7f0bb8886640 1 -- 172.21.15.42:0/3929812519 wait complete. 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.044+0000 7f0bb8886640 10 monclient: build_initial_monmap 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.044+0000 7f0bb8886640 10 monclient: monmap: 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:44.048 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.045+0000 7f0bb8886640 1 librados: starting msgr at 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.045+0000 7f0bb8886640 1 librados: starting objecter 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.045+0000 7f0bb8886640 1 Processor -- start 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.045+0000 7f0bb8886640 1 -- start start 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.045+0000 7f0bb8886640 1 librados: setting wanted keys 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.045+0000 7f0bb8886640 1 librados: calling monclient init 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.045+0000 7f0bb8886640 10 monclient: init 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 10 monclient(hunting): picked mon.noname-b con 0x7f0bb406c380 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 10 monclient(hunting): picked mon.noname-c con 0x7f0bb406d670 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 10 monclient(hunting): picked mon.noname-a con 0x7f0bb40af8f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb4144860 con 0x7f0bb40af8f0 2024-03-25T18:29:44.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb4144ad0 con 0x7f0bb406c380 2024-03-25T18:29:44.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb4144db0 con 0x7f0bb406d670 2024-03-25T18:29:44.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:44.050 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.046+0000 7f0bb8886640 10 monclient(hunting): authenticate will time out at 7899.273075s 2024-03-25T18:29:44.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.058+0000 7f0bb2d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0bb40af8f0 0x7f0bb41e2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56480/0 (socket says 172.21.15.42:56480) 2024-03-25T18:29:44.059 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.058+0000 7f0bb2d9d640 1 -- 172.21.15.42:0/2830929273 learned_addr learned my addr 172.21.15.42:0/2830929273 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:44.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.060+0000 7f0ba2ffd640 10 client.?.objecter ms_handle_connect 0x7f0bb40af8f0 2024-03-25T18:29:44.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.060+0000 7f0ba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:44.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.062+0000 7f0ba2ffd640 10 client.?.objecter ms_handle_connect 0x7f0bb406c380 2024-03-25T18:29:44.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.062+0000 7f0ba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:44.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.069+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177947250 0 0) 0x7f0bb4144860 con 0x7f0bb40af8f0 2024-03-25T18:29:44.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.069+0000 7f0ba2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:44.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.069+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:44.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.069+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:44.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.069+0000 7f0ba2ffd640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:29:44.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.069+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b94004eb0 con 0x7f0bb40af8f0 2024-03-25T18:29:44.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.070+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253535197 0 0) 0x7f0bb4144ad0 con 0x7f0bb406c380 2024-03-25T18:29:44.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.070+0000 7f0ba2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:44.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.070+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:44.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.070+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:44.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.070+0000 7f0ba2ffd640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:29:44.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.070+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb4144860 con 0x7f0bb406c380 2024-03-25T18:29:44.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.074+0000 7f0ba2ffd640 10 client.?.objecter ms_handle_connect 0x7f0bb406d670 2024-03-25T18:29:44.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.074+0000 7f0ba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.077+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3854749040 0 0) 0x7f0bb4144860 con 0x7f0bb406c380 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.077+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bb4144ad0 con 0x7f0bb406c380 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.077+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0ba4001930 con 0x7f0bb406c380 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.077+0000 7f0ba2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.077+0000 7f0ba2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.077+0000 7f0ba2ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:44.079 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:44.080 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:44.080 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:44.080 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:44.080 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:44.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.077+0000 7f0ba2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:44.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.088+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371377755 0 0) 0x7f0b94004eb0 con 0x7f0bb40af8f0 2024-03-25T18:29:44.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.088+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bb4144860 con 0x7f0bb40af8f0 2024-03-25T18:29:44.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.088+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0bac001c90 con 0x7f0bb40af8f0 2024-03-25T18:29:44.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.088+0000 7f0ba2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:44.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.088+0000 7f0ba2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:44.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.088+0000 7f0ba2ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:44.090 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.089+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920836981 0 0) 0x7f0bb4144db0 con 0x7f0bb406d670 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.089+0000 7f0ba2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.089+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:44.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.089+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:44.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.089+0000 7f0ba2ffd640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:29:44.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.089+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b94004eb0 con 0x7f0bb406d670 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3230470244 0 0) 0x7f0bb4144ad0 con 0x7f0bb406c380 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 >> v1:172.21.15.42:6790/0 conn(0x7f0bb406d670 legacy=0x7f0bb4143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 >> v1:172.21.15.42:6789/0 conn(0x7f0bb40af8f0 legacy=0x7f0bb41e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 1 monclient: found mon.b 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb4145090 con 0x7f0bb406c380 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0ba2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:29:44.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 10 monclient: _renew_subs 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb41454b0 con 0x7f0bb406c380 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 10 monclient: _renew_subs 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb4145760 con 0x7f0bb406c380 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.095+0000 7f0bb8886640 1 librados: init done 2024-03-25T18:29:44.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.096+0000 7f0bb8886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:44.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.096+0000 7f0bb8886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:44.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.096+0000 7f0bb8886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:44.099 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.096+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb40627b0 con 0x7f0bb406c380 2024-03-25T18:29:44.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.373+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba4003610 con 0x7f0bb406c380 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.374+0000 7f0ba2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.374+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f0ba4003960 con 0x7f0bb406c380 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.374+0000 7f0ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.374+0000 7f0ba2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.374+0000 7f0ba2ffd640 10 monclient: dump: 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:44.376 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.374+0000 7f0bb0d99640 4 set_mon_vals no callback set 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.374+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f0ba4003ac0 con 0x7f0bb406c380 2024-03-25T18:29:44.377 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.375+0000 7f0bb1d9b640 10 monclient: get_auth_request con 0x7f0b9406df50 auth_method 0 2024-03-25T18:29:44.378 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.375+0000 7f0ba2ffd640 10 client.4690.objecter ms_handle_connect 0x7f0b9406df50 2024-03-25T18:29:44.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.379+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7f0ba4053680 con 0x7f0bb406c380 2024-03-25T18:29:44.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.379+0000 7f0ba2ffd640 10 client.4690.objecter ms_dispatch 0x7f0bb4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:44.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.379+0000 7f0ba2ffd640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:44.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.379+0000 7f0ba2ffd640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:44.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.379+0000 7f0ba2ffd640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T18:29:44.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.380+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ba404f190 con 0x7f0bb406c380 2024-03-25T18:29:44.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.380+0000 7f0ba2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:44.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.380+0000 7f0ba2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:44.634 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.632+0000 7f0bb8886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:44.634 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.632+0000 7f0bb8886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:44.634 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.632+0000 7f0bb8886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:44.634 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.633+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bb40b64a0 con 0x7f0bb406c380 2024-03-25T18:29:44.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.784+0000 7f0ba2ffd640 1 -- 172.21.15.42:0/2830929273 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7f0ba40510c0 con 0x7f0bb406c380 2024-03-25T18:29:44.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.784+0000 7f0ba2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:44.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.784+0000 7f0ba2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:44.786 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:29:44.786 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:44.788 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.786+0000 7f0bb8886640 10 librados: watch_flush enter 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.786+0000 7f0bb8886640 10 librados: watch_flush exit 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.786+0000 7f0bb8886640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.786+0000 7f0bb8886640 10 client.4690.objecter successfully canceled tick 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 >> v1:172.21.15.42:6800/3081750888 conn(0x7f0b9406df50 legacy=0x7f0b94070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 10 monclient: shutdown 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 >> v1:172.21.15.120:6789/0 conn(0x7f0bb406c380 legacy=0x7f0bb41433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 shutdown_connections 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 >> 172.21.15.42:0/2830929273 conn(0x7f0bb40b5c00 msgr2=0x7f0bb4148510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 shutdown_connections 2024-03-25T18:29:44.789 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 1 -- 172.21.15.42:0/2830929273 wait complete. 2024-03-25T18:29:44.790 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:44.787+0000 7f0bb8886640 1 librados: shutdown 2024-03-25T18:29:44.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:51.609 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:51.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.724+0000 7f8d7b5d7640 10 monclient: get_monmap_and_config 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.724+0000 7f8d7b5d7640 10 monclient: build_initial_monmap 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.724+0000 7f8d7b5d7640 10 monclient: monmap: 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:51.725602+0000 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:51.725602+0000 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:51.726 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:51.727 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:51.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.725+0000 7f8d7b5d7640 1 Processor -- start 2024-03-25T18:29:51.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.725+0000 7f8d7b5d7640 1 -- start start 2024-03-25T18:29:51.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.725+0000 7f8d7b5d7640 10 monclient: init 2024-03-25T18:29:51.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 10 monclient(hunting): picked mon.noname-c con 0x7f8d740705b0 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 10 monclient(hunting): picked mon.noname-b con 0x7f8d741402d0 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 10 monclient(hunting): picked mon.noname-a con 0x7f8d74149ae0 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d740718c0 con 0x7f8d74149ae0 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d740714c0 con 0x7f8d740705b0 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d74072b50 con 0x7f8d741402d0 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:51.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.726+0000 7f8d7b5d7640 10 monclient(hunting): authenticate will time out at 7906.952834s 2024-03-25T18:29:51.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.735+0000 7f8d79b74640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8d74149ae0 0x7f8d7414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56996/0 (socket says 172.21.15.42:56996) 2024-03-25T18:29:51.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.735+0000 7f8d79b74640 1 -- 172.21.15.42:0/957190888 learned_addr learned my addr 172.21.15.42:0/957190888 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:51.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.753+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626966971 0 0) 0x7f8d740718c0 con 0x7f8d74149ae0 2024-03-25T18:29:51.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.753+0000 7f8d6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:51.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.753+0000 7f8d6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:51.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.753+0000 7f8d6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:51.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.753+0000 7f8d6bfff640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:29:51.755 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.753+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d60003520 con 0x7f8d74149ae0 2024-03-25T18:29:51.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.756+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233577880 0 0) 0x7f8d74072b50 con 0x7f8d741402d0 2024-03-25T18:29:51.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.756+0000 7f8d6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:51.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.756+0000 7f8d6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:51.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.756+0000 7f8d6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:51.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.756+0000 7f8d6bfff640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:29:51.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.756+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d740718c0 con 0x7f8d741402d0 2024-03-25T18:29:51.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.761+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557037410 0 0) 0x7f8d740714c0 con 0x7f8d740705b0 2024-03-25T18:29:51.762 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.761+0000 7f8d6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:51.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.761+0000 7f8d6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:51.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.761+0000 7f8d6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:51.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.761+0000 7f8d6bfff640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:29:51.763 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.761+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d74072b50 con 0x7f8d740705b0 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454508551 0 0) 0x7f8d60003520 con 0x7f8d74149ae0 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 >> v1:172.21.15.42:6790/0 conn(0x7f8d741402d0 legacy=0x7f8d74140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 >> v1:172.21.15.120:6789/0 conn(0x7f8d740705b0 legacy=0x7f8d7413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 1 monclient: found mon.noname-a 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7413dd10 con 0x7f8d74149ae0 2024-03-25T18:29:51.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 10 monclient: _finish_auth 0 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8d6c001a10 con 0x7f8d74149ae0 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d6bfff640 10 monclient: dump: 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:51.771 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:51.772 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:51.772 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:51.772 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:51.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d7b5d7640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:29:51.772 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.768+0000 7f8d7b5d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:51.779 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d6c001b70 con 0x7f8d74149ae0 2024-03-25T18:29:51.779 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:51.779 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d6bfff640 1 -- 172.21.15.42:0/957190888 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8d6c001f80 con 0x7f8d74149ae0 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d7b5d7640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d7b5d7640 4 set_mon_vals no callback set 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.778+0000 7f8d6bfff640 10 monclient: dump: 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:51.780 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:51.781 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:51.781 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:51.781 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:51.783 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.779+0000 7f8d7b5d7640 10 monclient: shutdown 2024-03-25T18:29:51.783 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.779+0000 7f8d7b5d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:51.783 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.779+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/957190888 >> v1:172.21.15.42:6789/0 conn(0x7f8d74149ae0 legacy=0x7f8d7414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.779+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/957190888 shutdown_connections 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.779+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/957190888 >> 172.21.15.42:0/957190888 conn(0x7f8d740b5d10 msgr2=0x7f8d740b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.779+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/957190888 shutdown_connections 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.779+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/957190888 wait complete. 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 10 monclient: build_initial_monmap 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 10 monclient: monmap: 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:51.784 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 1 librados: starting msgr at 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 1 librados: starting objecter 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 1 Processor -- start 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 1 -- start start 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 1 librados: setting wanted keys 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 1 librados: calling monclient init 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.780+0000 7f8d7b5d7640 10 monclient: init 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:51.785 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 10 monclient(hunting): picked mon.noname-c con 0x7f8d740705b0 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:51.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 10 monclient(hunting): picked mon.noname-a con 0x7f8d741402d0 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:51.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 10 monclient(hunting): picked mon.noname-b con 0x7f8d74149ae0 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:51.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d74144fc0 con 0x7f8d741402d0 2024-03-25T18:29:51.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d74145230 con 0x7f8d74149ae0 2024-03-25T18:29:51.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d741d6040 con 0x7f8d740705b0 2024-03-25T18:29:51.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:51.786 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.781+0000 7f8d7b5d7640 10 monclient(hunting): authenticate will time out at 7907.008311s 2024-03-25T18:29:51.792 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.791+0000 7f8d79b74640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8d74149ae0 0x7f8d741dfd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:50614/0 (socket says 172.21.15.42:50614) 2024-03-25T18:29:51.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.791+0000 7f8d79b74640 1 -- 172.21.15.42:0/1660728018 learned_addr learned my addr 172.21.15.42:0/1660728018 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:51.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.792+0000 7f8d69ffb640 10 client.?.objecter ms_handle_connect 0x7f8d74149ae0 2024-03-25T18:29:51.793 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.792+0000 7f8d69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:51.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.796+0000 7f8d69ffb640 10 client.?.objecter ms_handle_connect 0x7f8d741402d0 2024-03-25T18:29:51.797 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.796+0000 7f8d69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:51.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.798+0000 7f8d69ffb640 10 client.?.objecter ms_handle_connect 0x7f8d740705b0 2024-03-25T18:29:51.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.798+0000 7f8d69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:51.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.801+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106073916 0 0) 0x7f8d74145230 con 0x7f8d74149ae0 2024-03-25T18:29:51.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.801+0000 7f8d69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:51.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.801+0000 7f8d69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:51.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.801+0000 7f8d69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:51.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.801+0000 7f8d69ffb640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:29:51.803 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.801+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d58004eb0 con 0x7f8d74149ae0 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.808+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480374780 0 0) 0x7f8d58004eb0 con 0x7f8d74149ae0 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.808+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d74145230 con 0x7f8d74149ae0 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.808+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8d6c001e80 con 0x7f8d74149ae0 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.808+0000 7f8d69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.808+0000 7f8d69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.808+0000 7f8d69ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:51.809 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:51.810 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.808+0000 7f8d69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:51.812 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.811+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215034433 0 0) 0x7f8d74144fc0 con 0x7f8d741402d0 2024-03-25T18:29:51.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.811+0000 7f8d69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:51.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.811+0000 7f8d69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:51.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.811+0000 7f8d69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:51.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.811+0000 7f8d69ffb640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:29:51.813 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.811+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d58004eb0 con 0x7f8d741402d0 2024-03-25T18:29:51.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.813+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424228324 0 0) 0x7f8d741d6040 con 0x7f8d740705b0 2024-03-25T18:29:51.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.813+0000 7f8d69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:51.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.813+0000 7f8d69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:51.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.813+0000 7f8d69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:51.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.813+0000 7f8d69ffb640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:29:51.814 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.813+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d74144fc0 con 0x7f8d740705b0 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1835620042 0 0) 0x7f8d74145230 con 0x7f8d74149ae0 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 >> v1:172.21.15.42:6790/0 conn(0x7f8d740705b0 legacy=0x7f8d741d52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 >> v1:172.21.15.42:6789/0 conn(0x7f8d741402d0 legacy=0x7f8d741d87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 1 monclient: found mon.b 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d741d8ef0 con 0x7f8d74149ae0 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:51.818 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 10 monclient: _renew_subs 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d741d91a0 con 0x7f8d74149ae0 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 10 monclient: _renew_subs 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7413f900 con 0x7f8d74149ae0 2024-03-25T18:29:51.819 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.816+0000 7f8d7b5d7640 1 librados: init done 2024-03-25T18:29:51.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.818+0000 7f8d7b5d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:51.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.818+0000 7f8d7b5d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:51.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.818+0000 7f8d7b5d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:51.820 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:51.818+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d740627b0 con 0x7f8d74149ae0 2024-03-25T18:29:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.091+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d6c0020c0 con 0x7f8d74149ae0 2024-03-25T18:29:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.091+0000 7f8d69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:52.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.091+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8d6c004fb0 con 0x7f8d74149ae0 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.091+0000 7f8d69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.091+0000 7f8d69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.091+0000 7f8d69ffb640 10 monclient: dump: 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:52.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.091+0000 7f8d6b7fe640 4 set_mon_vals no callback set 2024-03-25T18:29:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.092+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f8d6c01af80 con 0x7f8d74149ae0 2024-03-25T18:29:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.092+0000 7f8d79373640 10 monclient: get_auth_request con 0x7f8d5806de80 auth_method 0 2024-03-25T18:29:52.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.092+0000 7f8d69ffb640 10 client.4696.objecter ms_handle_connect 0x7f8d5806de80 2024-03-25T18:29:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7f8d6c054c00 con 0x7f8d74149ae0 2024-03-25T18:29:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 10 client.4696.objecter ms_dispatch 0x7f8d74000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 3 client.4696.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 3 client.4696.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T18:29:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d6c050540 con 0x7f8d74149ae0 2024-03-25T18:29:52.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:52.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.096+0000 7f8d69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:52.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.333+0000 7f8d7b5d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:52.334 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.333+0000 7f8d7b5d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:52.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.333+0000 7f8d7b5d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:52.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.333+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d74069f40 con 0x7f8d74149ae0 2024-03-25T18:29:52.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.390+0000 7f8d69ffb640 1 -- 172.21.15.42:0/1660728018 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7f8d6c052640 con 0x7f8d74149ae0 2024-03-25T18:29:52.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.390+0000 7f8d69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:52.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.390+0000 7f8d69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:52.392 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:29:52.392 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 10 librados: watch_flush enter 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 10 librados: watch_flush exit 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 10 client.4696.objecter successfully canceled tick 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 >> v1:172.21.15.42:6800/3081750888 conn(0x7f8d5806de80 legacy=0x7f8d58070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 10 monclient: shutdown 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:52.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 >> v1:172.21.15.120:6789/0 conn(0x7f8d74149ae0 legacy=0x7f8d741dfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:52.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 shutdown_connections 2024-03-25T18:29:52.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.392+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 >> 172.21.15.42:0/1660728018 conn(0x7f8d740b5d10 msgr2=0x7f8d7413c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:52.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.393+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 shutdown_connections 2024-03-25T18:29:52.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.393+0000 7f8d7b5d7640 1 -- 172.21.15.42:0/1660728018 wait complete. 2024-03-25T18:29:52.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:52.393+0000 7f8d7b5d7640 1 librados: shutdown 2024-03-25T18:29:52.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:53.501 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:36.670 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.555 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:36.724 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.585 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:36.754 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.611 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:36.731 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.621 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:36.742 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.623 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:36.792 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.633 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:36.802 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.659 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:36.779 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:53.678 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:36.847 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:59.214 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:59.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.325+0000 7f22d8939640 10 monclient: get_monmap_and_config 2024-03-25T18:29:59.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.325+0000 7f22d8939640 10 monclient: build_initial_monmap 2024-03-25T18:29:59.326 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.325+0000 7f22d8939640 10 monclient: monmap: 2024-03-25T18:29:59.326 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:29:59.326302+0000 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:29:59.326302+0000 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:29:59.327 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 1 Processor -- start 2024-03-25T18:29:59.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 1 -- start start 2024-03-25T18:29:59.328 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient: init 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient(hunting): picked mon.noname-b con 0x7f22d40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient(hunting): picked mon.noname-a con 0x7f22d413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient(hunting): picked mon.noname-c con 0x7f22d41430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d4142520 con 0x7f22d413f710 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d40b28a0 con 0x7f22d41430c0 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d40b2a30 con 0x7f22d40a98f0 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:59.329 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.326+0000 7f22d8939640 10 monclient(hunting): authenticate will time out at 7914.553502s 2024-03-25T18:29:59.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.334+0000 7f22d2d9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f22d41430c0 0x7f22d414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:54252/0 (socket says 172.21.15.42:54252) 2024-03-25T18:29:59.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.334+0000 7f22d2d9d640 1 -- 172.21.15.42:0/1288580233 learned_addr learned my addr 172.21.15.42:0/1288580233 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:59.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.348+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759249390 0 0) 0x7f22d40b28a0 con 0x7f22d41430c0 2024-03-25T18:29:59.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.348+0000 7f22d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:59.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.348+0000 7f22d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:59.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.348+0000 7f22d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:59.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.348+0000 7f22d1d9b640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:29:59.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.348+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c0003520 con 0x7f22d41430c0 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2176350380 0 0) 0x7f22c0003520 con 0x7f22d41430c0 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 >> v1:172.21.15.42:6790/0 conn(0x7f22d40a98f0 legacy=0x7f22d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 >> v1:172.21.15.42:6789/0 conn(0x7f22d413f710 legacy=0x7f22d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d40b44c0 con 0x7f22d41430c0 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22cc001a10 con 0x7f22d41430c0 2024-03-25T18:29:59.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d1d9b640 10 monclient: dump: 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.358 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d8939640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:29:59.359 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.355+0000 7f22d8939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:59.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22cc001b70 con 0x7f22d41430c0 2024-03-25T18:29:59.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:59.369 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d1d9b640 1 -- 172.21.15.42:0/1288580233 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22cc001fc0 con 0x7f22d41430c0 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d1d9b640 10 monclient: dump: 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:59.370 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:59.371 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:59.371 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:59.371 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d8939640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:59.371 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.368+0000 7f22d8939640 4 set_mon_vals no callback set 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.369+0000 7f22d8939640 10 monclient: shutdown 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.369+0000 7f22d8939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.369+0000 7f22d8939640 1 -- 172.21.15.42:0/1288580233 >> v1:172.21.15.120:6789/0 conn(0x7f22d41430c0 legacy=0x7f22d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.369+0000 7f22d8939640 1 -- 172.21.15.42:0/1288580233 shutdown_connections 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.369+0000 7f22d8939640 1 -- 172.21.15.42:0/1288580233 >> 172.21.15.42:0/1288580233 conn(0x7f22d40b7ea0 msgr2=0x7f22d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.369+0000 7f22d8939640 1 -- 172.21.15.42:0/1288580233 shutdown_connections 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.369+0000 7f22d8939640 1 -- 172.21.15.42:0/1288580233 wait complete. 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 10 monclient: build_initial_monmap 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 10 monclient: monmap: 2024-03-25T18:29:59.373 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 1 librados: starting msgr at 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 1 librados: starting objecter 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 1 Processor -- start 2024-03-25T18:29:59.374 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 1 -- start start 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 1 librados: setting wanted keys 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 1 librados: calling monclient init 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.370+0000 7f22d8939640 10 monclient: init 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 10 monclient(hunting): picked mon.noname-c con 0x7f22d40a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 10 monclient(hunting): picked mon.noname-a con 0x7f22d413f710 addr v1:172.21.15.42:6789/0 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 10 monclient(hunting): picked mon.noname-b con 0x7f22d41430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d4137ab0 con 0x7f22d413f710 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d4138bd0 con 0x7f22d41430c0 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d4138e40 con 0x7f22d40a98f0 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:59.375 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.371+0000 7f22d8939640 10 monclient(hunting): authenticate will time out at 7914.598165s 2024-03-25T18:29:59.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.389+0000 7f22cbfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f22d413f710 0x7f22d41df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42516/0 (socket says 172.21.15.42:42516) 2024-03-25T18:29:59.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.389+0000 7f22cbfff640 1 -- 172.21.15.42:0/1270595129 learned_addr learned my addr 172.21.15.42:0/1270595129 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:29:59.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.390+0000 7f22caffd640 10 client.?.objecter ms_handle_connect 0x7f22d413f710 2024-03-25T18:29:59.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.391+0000 7f22caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:59.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.391+0000 7f22caffd640 10 client.?.objecter ms_handle_connect 0x7f22d40a98f0 2024-03-25T18:29:59.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.391+0000 7f22caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:59.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.395+0000 7f22caffd640 10 client.?.objecter ms_handle_connect 0x7f22d41430c0 2024-03-25T18:29:59.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.395+0000 7f22caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:59.409 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.408+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180037053 0 0) 0x7f22d4137ab0 con 0x7f22d413f710 2024-03-25T18:29:59.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.408+0000 7f22caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:59.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.408+0000 7f22caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:59.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.408+0000 7f22caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:59.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.408+0000 7f22caffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:29:59.410 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.408+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b4004eb0 con 0x7f22d413f710 2024-03-25T18:29:59.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.412+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598531985 0 0) 0x7f22d4138bd0 con 0x7f22d41430c0 2024-03-25T18:29:59.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.412+0000 7f22caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:59.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.412+0000 7f22caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:59.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.412+0000 7f22caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:59.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.412+0000 7f22caffd640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:29:59.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.412+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d4137ab0 con 0x7f22d41430c0 2024-03-25T18:29:59.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.413+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909163189 0 0) 0x7f22d4138e40 con 0x7f22d40a98f0 2024-03-25T18:29:59.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.413+0000 7f22caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:59.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.413+0000 7f22caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:59.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.413+0000 7f22caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:59.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.413+0000 7f22caffd640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:29:59.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.413+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d4138bd0 con 0x7f22d40a98f0 2024-03-25T18:29:59.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.431+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4292795359 0 0) 0x7f22d4138bd0 con 0x7f22d40a98f0 2024-03-25T18:29:59.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.431+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22d4138e40 con 0x7f22d40a98f0 2024-03-25T18:29:59.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.431+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22c4001830 con 0x7f22d40a98f0 2024-03-25T18:29:59.432 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.431+0000 7f22caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.431+0000 7f22caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.431+0000 7f22caffd640 10 monclient(hunting): dump: 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.433 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.431+0000 7f22caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.432+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717061514 0 0) 0x7f22b4004eb0 con 0x7f22d413f710 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.432+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22d4138bd0 con 0x7f22d413f710 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.432+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22bc0019d0 con 0x7f22d413f710 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.432+0000 7f22caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.432+0000 7f22caffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.432+0000 7f22caffd640 10 monclient(hunting): dump: 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:59.434 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:59.435 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.434+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712045626 0 0) 0x7f22d4137ab0 con 0x7f22d41430c0 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.434+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22b4004eb0 con 0x7f22d41430c0 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.434+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22cc001e80 con 0x7f22d41430c0 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.434+0000 7f22caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.434+0000 7f22caffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.434+0000 7f22caffd640 10 monclient(hunting): dump: 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:59.436 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:59.437 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:59.437 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:59.437 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.439+0000 7f22cbfff640 1 -- 172.21.15.42:0/1270595129 >> v1:172.21.15.42:6789/0 conn(0x7f22d413f710 legacy=0x7f22d41df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:29:59.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.439+0000 7f22cbfff640 1 -- 172.21.15.42:0/1270595129 >> v1:172.21.15.42:6789/0 conn(0x7f22d413f710 legacy=0x7f22d41df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:29:59.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.439+0000 7f22cbfff640 1 --1- 172.21.15.42:0/1270595129 >> v1:172.21.15.42:6789/0 conn(0x7f22d413f710 0x7f22d41df270 :-1 s=OPENED pgs=75967 cs=1 l=1).handle_message read tag failed 2024-03-25T18:29:59.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.439+0000 7f22cbfff640 1 --1- 172.21.15.42:0/1270595129 >> v1:172.21.15.42:6789/0 conn(0x7f22d413f710 0x7f22d41df270 :-1 s=OPENED pgs=75967 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:29:59.440 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.439+0000 7f22caffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.42:6789/0 2024-03-25T18:29:59.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 710088099 0 0) 0x7f22b4004eb0 con 0x7f22d41430c0 2024-03-25T18:29:59.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 >> v1:172.21.15.42:6790/0 conn(0x7f22d40a98f0 legacy=0x7f22d4137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:59.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 >> v1:172.21.15.42:6789/0 conn(0x7f22d413f710 legacy=0x7f22d41df270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:29:59.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:59.449 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 1 monclient: found mon.b 2024-03-25T18:29:59.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:59.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:59.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4139120 con 0x7f22d41430c0 2024-03-25T18:29:59.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:59.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:59.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:59.450 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 10 monclient: _renew_subs 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d4139400 con 0x7f22d41430c0 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 10 monclient: _renew_subs 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d40b7880 con 0x7f22d41430c0 2024-03-25T18:29:59.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.447+0000 7f22d8939640 1 librados: init done 2024-03-25T18:29:59.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.448+0000 7f22d8939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:59.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.448+0000 7f22d8939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:59.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.448+0000 7f22d8939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:59.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.448+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d4137810 con 0x7f22d41430c0 2024-03-25T18:29:59.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.453+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22cc002040 con 0x7f22d41430c0 2024-03-25T18:29:59.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.453+0000 7f22caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:59.454 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.453+0000 7f22d159a640 4 set_mon_vals no callback set 2024-03-25T18:29:59.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.721+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f22cc004fb0 con 0x7f22d41430c0 2024-03-25T18:29:59.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.721+0000 7f22caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:59.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.721+0000 7f22caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:59.723 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.721+0000 7f22caffd640 10 monclient: dump: 2024-03-25T18:29:59.723 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:29:59.723 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:29:59.723 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.724 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:29:59.724 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:59.724 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:29:59.724 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:29:59.724 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:29:59.724 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:29:59.724 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:29:59.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.722+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f22cc01af80 con 0x7f22d41430c0 2024-03-25T18:29:59.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.723+0000 7f22d259c640 10 monclient: get_auth_request con 0x7f22b406e0d0 auth_method 0 2024-03-25T18:29:59.725 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.723+0000 7f22caffd640 10 client.4702.objecter ms_handle_connect 0x7f22b406e0d0 2024-03-25T18:29:59.727 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.726+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7f22cc054a80 con 0x7f22d41430c0 2024-03-25T18:29:59.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.726+0000 7f22caffd640 10 client.4702.objecter ms_dispatch 0x7f22d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:59.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.726+0000 7f22caffd640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:59.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.726+0000 7f22caffd640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:59.728 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.726+0000 7f22caffd640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T18:29:59.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.731+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22cc050590 con 0x7f22d41430c0 2024-03-25T18:29:59.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.731+0000 7f22caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:59.732 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.731+0000 7f22caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:59.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.979+0000 7f22d8939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:59.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.979+0000 7f22d8939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:59.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.979+0000 7f22d8939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:29:59.981 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:29:59.979+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22d4137c60 con 0x7f22d41430c0 2024-03-25T18:30:00.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.124+0000 7f22caffd640 1 -- 172.21.15.42:0/1270595129 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7f22d4137c60 con 0x7f22d41430c0 2024-03-25T18:30:00.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.124+0000 7f22caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:00.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.124+0000 7f22caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:00.126 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:00.126 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:00.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 10 librados: watch_flush enter 2024-03-25T18:30:00.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 10 librados: watch_flush exit 2024-03-25T18:30:00.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T18:30:00.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 10 client.4702.objecter successfully canceled tick 2024-03-25T18:30:00.128 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 >> v1:172.21.15.42:6800/3081750888 conn(0x7f22b406e0d0 legacy=0x7f22b4070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 10 monclient: shutdown 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.126+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 >> v1:172.21.15.120:6789/0 conn(0x7f22d41430c0 legacy=0x7f22d41e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.127+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 shutdown_connections 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.127+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 >> 172.21.15.42:0/1270595129 conn(0x7f22d40b7ea0 msgr2=0x7f22d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.127+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 shutdown_connections 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.127+0000 7f22d8939640 1 -- 172.21.15.42:0/1270595129 wait complete. 2024-03-25T18:30:00.129 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:00.127+0000 7f22d8939640 1 librados: shutdown 2024-03-25T18:30:00.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:06.948 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.059+0000 7f663eb82640 10 monclient: get_monmap_and_config 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.059+0000 7f663eb82640 10 monclient: build_initial_monmap 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.059+0000 7f663eb82640 10 monclient: monmap: 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:30:07.060808+0000 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:30:07.060808+0000 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:30:07.061 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:30:07.062 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:07.062 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.060+0000 7f663eb82640 1 Processor -- start 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.060+0000 7f663eb82640 1 -- start start 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.060+0000 7f663eb82640 10 monclient: init 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 10 monclient(hunting): picked mon.noname-a con 0x7f66380a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 10 monclient(hunting): picked mon.noname-c con 0x7f663813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 10 monclient(hunting): picked mon.noname-b con 0x7f66381430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6638142520 con 0x7f66380a98f0 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66380b28a0 con 0x7f663813f710 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66380b2a30 con 0x7f66381430c0 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:07.063 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.061+0000 7f663eb82640 10 monclient(hunting): authenticate will time out at 7922.287913s 2024-03-25T18:30:07.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.069+0000 7f6637fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f663813f710 0x7f66381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:58918/0 (socket says 172.21.15.42:58918) 2024-03-25T18:30:07.070 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.069+0000 7f6637fff640 1 -- 172.21.15.42:0/213166265 learned_addr learned my addr 172.21.15.42:0/213166265 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:07.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.078+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271206884 0 0) 0x7f66380b28a0 con 0x7f663813f710 2024-03-25T18:30:07.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.078+0000 7f66377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:07.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.078+0000 7f66377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:07.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.078+0000 7f66377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:07.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.078+0000 7f66377fe640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:30:07.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.078+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6628003520 con 0x7f663813f710 2024-03-25T18:30:07.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.084+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442232826 0 0) 0x7f6638142520 con 0x7f66380a98f0 2024-03-25T18:30:07.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.084+0000 7f66377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:07.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.084+0000 7f66377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:07.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.084+0000 7f66377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:07.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.084+0000 7f66377fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:30:07.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.084+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66380b28a0 con 0x7f66380a98f0 2024-03-25T18:30:07.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.089+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424581547 0 0) 0x7f66380b2a30 con 0x7f66381430c0 2024-03-25T18:30:07.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.089+0000 7f66377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:07.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.089+0000 7f66377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:07.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.089+0000 7f66377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:07.090 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.089+0000 7f66377fe640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:30:07.091 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.089+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638142520 con 0x7f66381430c0 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 715639254 0 0) 0x7f6628003520 con 0x7f663813f710 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 >> v1:172.21.15.42:6790/0 conn(0x7f66381430c0 legacy=0x7f663814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 >> v1:172.21.15.42:6789/0 conn(0x7f66380a98f0 legacy=0x7f66380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 1 monclient: found mon.noname-c 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380b44c0 con 0x7f663813f710 2024-03-25T18:30:07.094 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6624001690 con 0x7f663813f710 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f663eb82640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f663eb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f66377fe640 10 monclient: dump: 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:07.095 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:07.096 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:07.096 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:07.096 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:07.096 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:07.096 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.092+0000 7f663eb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:07.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.103+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66240017f0 con 0x7f663813f710 2024-03-25T18:30:07.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.103+0000 7f66377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:07.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.103+0000 7f66377fe640 1 -- 172.21.15.42:0/213166265 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6624001c00 con 0x7f663813f710 2024-03-25T18:30:07.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f66377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:07.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f66377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f66377fe640 10 monclient: dump: 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:07.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 4 set_mon_vals no callback set 2024-03-25T18:30:07.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 10 monclient: shutdown 2024-03-25T18:30:07.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:07.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 1 -- 172.21.15.42:0/213166265 >> v1:172.21.15.120:6789/0 conn(0x7f663813f710 legacy=0x7f66381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:07.108 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 1 -- 172.21.15.42:0/213166265 shutdown_connections 2024-03-25T18:30:07.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 1 -- 172.21.15.42:0/213166265 >> 172.21.15.42:0/213166265 conn(0x7f66380b7ea0 msgr2=0x7f66380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:07.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.104+0000 7f663eb82640 1 -- 172.21.15.42:0/213166265 shutdown_connections 2024-03-25T18:30:07.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.105+0000 7f663eb82640 1 -- 172.21.15.42:0/213166265 wait complete. 2024-03-25T18:30:07.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.105+0000 7f663eb82640 10 monclient: build_initial_monmap 2024-03-25T18:30:07.109 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.105+0000 7f663eb82640 10 monclient: monmap: 2024-03-25T18:30:07.109 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:07.109 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 1 librados: starting msgr at 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 1 librados: starting objecter 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 1 Processor -- start 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 1 -- start start 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 1 librados: setting wanted keys 2024-03-25T18:30:07.110 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 1 librados: calling monclient init 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 10 monclient: init 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.106+0000 7f663eb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 10 monclient(hunting): picked mon.noname-a con 0x7f66380a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 10 monclient(hunting): picked mon.noname-c con 0x7f663813f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 10 monclient(hunting): picked mon.noname-b con 0x7f66381430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6638138c10 con 0x7f66380a98f0 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6638138e60 con 0x7f66381430c0 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6638139140 con 0x7f663813f710 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:07.111 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.107+0000 7f663eb82640 10 monclient(hunting): authenticate will time out at 7922.333739s 2024-03-25T18:30:07.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.115+0000 7f663c91e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f66380a98f0 0x7f6638137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47436/0 (socket says 172.21.15.42:47436) 2024-03-25T18:30:07.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.115+0000 7f663c91e640 1 -- 172.21.15.42:0/1697335897 learned_addr learned my addr 172.21.15.42:0/1697335897 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:07.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.115+0000 7f66357fa640 10 client.?.objecter ms_handle_connect 0x7f66380a98f0 2024-03-25T18:30:07.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.115+0000 7f66357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:07.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.117+0000 7f66357fa640 10 client.?.objecter ms_handle_connect 0x7f66381430c0 2024-03-25T18:30:07.118 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.117+0000 7f66357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:07.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.123+0000 7f66357fa640 10 client.?.objecter ms_handle_connect 0x7f663813f710 2024-03-25T18:30:07.124 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.123+0000 7f66357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:07.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.124+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292945336 0 0) 0x7f6638138e60 con 0x7f66381430c0 2024-03-25T18:30:07.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.124+0000 7f66357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:07.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.124+0000 7f66357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:07.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.124+0000 7f66357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:07.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.124+0000 7f66357fa640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:30:07.125 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.124+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c004eb0 con 0x7f66381430c0 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4257989378 0 0) 0x7f661c004eb0 con 0x7f66381430c0 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6638138e60 con 0x7f66381430c0 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6630001bd0 con 0x7f66381430c0 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): dump: 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.135 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881010943 0 0) 0x7f6638139140 con 0x7f663813f710 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:30:07.136 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.133+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c004eb0 con 0x7f663813f710 2024-03-25T18:30:07.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.134+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592651952 0 0) 0x7f6638138c10 con 0x7f66380a98f0 2024-03-25T18:30:07.137 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.134+0000 7f66357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:07.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.134+0000 7f66357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:07.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.134+0000 7f66357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:07.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.134+0000 7f66357fa640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:30:07.138 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.134+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638139140 con 0x7f66380a98f0 2024-03-25T18:30:07.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.146+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1501284223 0 0) 0x7f6638138e60 con 0x7f66381430c0 2024-03-25T18:30:07.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 >> v1:172.21.15.42:6790/0 conn(0x7f663813f710 legacy=0x7f6638132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:07.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 >> v1:172.21.15.42:6789/0 conn(0x7f66380a98f0 legacy=0x7f6638137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:07.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:07.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 1 monclient: found mon.b 2024-03-25T18:30:07.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:07.149 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638139920 con 0x7f66381430c0 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f66357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 10 monclient: _renew_subs 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66381382e0 con 0x7f66381430c0 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 10 monclient: _renew_subs 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66380ab8b0 con 0x7f66381430c0 2024-03-25T18:30:07.150 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.147+0000 7f663eb82640 1 librados: init done 2024-03-25T18:30:07.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.148+0000 7f663eb82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:07.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.148+0000 7f663eb82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:07.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.148+0000 7f663eb82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:07.151 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.148+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6638137a20 con 0x7f66381430c0 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.165+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66300020a0 con 0x7f66381430c0 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.165+0000 7f66357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.165+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6630003b30 con 0x7f66381430c0 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.165+0000 7f66357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.165+0000 7f66357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.165+0000 7f66357fa640 10 monclient: dump: 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:07.167 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:07.168 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:07.168 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:07.168 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:07.168 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:07.168 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:07.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.165+0000 7f6636ffd640 4 set_mon_vals no callback set 2024-03-25T18:30:07.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.166+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f6630003d50 con 0x7f66381430c0 2024-03-25T18:30:07.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.167+0000 7f663c91e640 10 monclient: get_auth_request con 0x7f661c06de10 auth_method 0 2024-03-25T18:30:07.169 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.167+0000 7f66357fa640 10 client.4708.objecter ms_handle_connect 0x7f661c06de10 2024-03-25T18:30:07.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.168+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7f66300539a0 con 0x7f66381430c0 2024-03-25T18:30:07.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.168+0000 7f66357fa640 10 client.4708.objecter ms_dispatch 0x7f6638000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:07.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.168+0000 7f66357fa640 3 client.4708.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:07.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.168+0000 7f66357fa640 3 client.4708.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:07.170 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.168+0000 7f66357fa640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T18:30:07.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.426+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f663004f2e0 con 0x7f66381430c0 2024-03-25T18:30:07.427 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.426+0000 7f66357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:07.428 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.426+0000 7f66357fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:07.663 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.661+0000 7f663eb82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:07.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.661+0000 7f663eb82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:07.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.661+0000 7f663eb82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:07.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.661+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f663813d010 con 0x7f66381430c0 2024-03-25T18:30:07.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.677+0000 7f66357fa640 1 -- 172.21.15.42:0/1697335897 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7f66300513e0 con 0x7f66381430c0 2024-03-25T18:30:07.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.678+0000 7f66357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:07.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.678+0000 7f66357fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:07.680 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:07.680 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:07.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 10 librados: watch_flush enter 2024-03-25T18:30:07.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 10 librados: watch_flush exit 2024-03-25T18:30:07.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T18:30:07.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 10 client.4708.objecter successfully canceled tick 2024-03-25T18:30:07.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 >> v1:172.21.15.42:6800/3081750888 conn(0x7f661c06de10 legacy=0x7f661c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:07.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 10 monclient: shutdown 2024-03-25T18:30:07.682 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:07.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 >> v1:172.21.15.120:6789/0 conn(0x7f66381430c0 legacy=0x7f66381e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:07.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 shutdown_connections 2024-03-25T18:30:07.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 >> 172.21.15.42:0/1697335897 conn(0x7f66380b7ea0 msgr2=0x7f6638074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:07.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 shutdown_connections 2024-03-25T18:30:07.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.680+0000 7f663eb82640 1 -- 172.21.15.42:0/1697335897 wait complete. 2024-03-25T18:30:07.683 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:07.681+0000 7f663eb82640 1 librados: shutdown 2024-03-25T18:30:07.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:13.565 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:56.734 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.578 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:56.747 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.613 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:10:56.782 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.620 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:56.740 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.624 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:56.744 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.629 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:56.798 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.632 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:56.802 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.647 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:10:56.767 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:13.654 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:10:56.824 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:14.502 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.603+0000 7ffa52d3d640 10 monclient: get_monmap_and_config 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.603+0000 7ffa52d3d640 10 monclient: build_initial_monmap 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.603+0000 7ffa52d3d640 10 monclient: monmap: 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:30:14.603848+0000 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:30:14.603848+0000 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:14.604 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:14.605 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:30:14.605 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:30:14.605 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.604+0000 7ffa52d3d640 1 Processor -- start 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.604+0000 7ffa52d3d640 1 -- start start 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.604+0000 7ffa52d3d640 10 monclient: init 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c0b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4c0b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:14.606 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c14be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:14.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0af940 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0b0a70 con 0x7ffa4c14be90 2024-03-25T18:30:14.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0b0d50 con 0x7ffa4c0b1730 2024-03-25T18:30:14.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:14.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa52d3d640 10 monclient(hunting): authenticate will time out at 7929.830736s 2024-03-25T18:30:14.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa4bfff640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7ffa4c0b1730 0x7ffa4c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:59424/0 (socket says 172.21.15.42:59424) 2024-03-25T18:30:14.607 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.605+0000 7ffa4bfff640 1 -- 172.21.15.42:0/377341303 learned_addr learned my addr 172.21.15.42:0/377341303 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:14.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.645+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836373631 0 0) 0x7ffa4c0af940 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.645+0000 7ffa4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:14.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.645+0000 7ffa4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:14.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.645+0000 7ffa4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:14.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.645+0000 7ffa4b7fe640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:30:14.646 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.645+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa3c003520 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.664 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62064479 0 0) 0x7ffa3c003520 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 >> v1:172.21.15.42:6790/0 conn(0x7ffa4c0b1730 legacy=0x7ffa4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 >> v1:172.21.15.120:6789/0 conn(0x7ffa4c14be90 legacy=0x7ffa4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.42:6789/0 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0b1030 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ffa40001690 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:14.665 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa4b7fe640 10 monclient: dump: 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa52d3d640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:30:14.666 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.664+0000 7ffa52d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:14.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 <== mon.0 v1:172.21.15.42:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa400017f0 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:14.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa4b7fe640 1 -- 172.21.15.42:0/377341303 <== mon.0 v1:172.21.15.42:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ffa40001c40 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:14.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 4 set_mon_vals no callback set 2024-03-25T18:30:14.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:14.687 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa4b7fe640 10 monclient: dump: 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:14.688 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:14.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 10 monclient: shutdown 2024-03-25T18:30:14.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:14.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 1 -- 172.21.15.42:0/377341303 >> v1:172.21.15.42:6789/0 conn(0x7ffa4c0b4d60 legacy=0x7ffa4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.690 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 1 -- 172.21.15.42:0/377341303 shutdown_connections 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 1 -- 172.21.15.42:0/377341303 >> 172.21.15.42:0/377341303 conn(0x7ffa4c0617a0 msgr2=0x7ffa4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 1 -- 172.21.15.42:0/377341303 shutdown_connections 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.687+0000 7ffa52d3d640 1 -- 172.21.15.42:0/377341303 wait complete. 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.688+0000 7ffa52d3d640 10 monclient: build_initial_monmap 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.688+0000 7ffa52d3d640 10 monclient: monmap: 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:30:14.691 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.688+0000 7ffa52d3d640 1 librados: starting msgr at 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.688+0000 7ffa52d3d640 1 librados: starting objecter 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.689+0000 7ffa52d3d640 1 Processor -- start 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.689+0000 7ffa52d3d640 1 -- start start 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.689+0000 7ffa52d3d640 1 librados: setting wanted keys 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.689+0000 7ffa52d3d640 1 librados: calling monclient init 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.689+0000 7ffa52d3d640 10 monclient: init 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c0b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4c0b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c14be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c1368f0 con 0x7ffa4c0b1730 2024-03-25T18:30:14.692 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c137750 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c1363d0 con 0x7ffa4c14be90 2024-03-25T18:30:14.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:14.693 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.690+0000 7ffa52d3d640 10 monclient(hunting): authenticate will time out at 7929.915979s 2024-03-25T18:30:14.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.701+0000 7ffa4bfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ffa4c0b4d60 0x7ffa4c12fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:40720/0 (socket says 172.21.15.42:40720) 2024-03-25T18:30:14.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.701+0000 7ffa4bfff640 1 -- 172.21.15.42:0/2737516455 learned_addr learned my addr 172.21.15.42:0/2737516455 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:14.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.702+0000 7ffa497fa640 10 client.?.objecter ms_handle_connect 0x7ffa4c0b4d60 2024-03-25T18:30:14.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.702+0000 7ffa497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:14.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.705+0000 7ffa497fa640 10 client.?.objecter ms_handle_connect 0x7ffa4c14be90 2024-03-25T18:30:14.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.705+0000 7ffa497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:14.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425945810 0 0) 0x7ffa4c137750 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:14.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:14.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:14.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:30:14.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa30004eb0 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.712 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 10 client.?.objecter ms_handle_connect 0x7ffa4c0b1730 2024-03-25T18:30:14.713 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.712+0000 7ffa497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:14.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.733+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836959793 0 0) 0x7ffa4c1363d0 con 0x7ffa4c14be90 2024-03-25T18:30:14.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.733+0000 7ffa497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:14.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.733+0000 7ffa497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:14.733 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.733+0000 7ffa497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:14.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.733+0000 7ffa497fa640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:30:14.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.733+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c137750 con 0x7ffa4c14be90 2024-03-25T18:30:14.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.734+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358318390 0 0) 0x7ffa30004eb0 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.734+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4c1363d0 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.734+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ffa380018b0 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.734+0000 7ffa497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:14.734 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.734+0000 7ffa497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.734+0000 7ffa497fa640 10 monclient(hunting): dump: 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:14.735 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.734+0000 7ffa497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:14.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.739+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395172993 0 0) 0x7ffa4c1368f0 con 0x7ffa4c0b1730 2024-03-25T18:30:14.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.739+0000 7ffa497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:14.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.739+0000 7ffa497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:14.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.739+0000 7ffa497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:14.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.739+0000 7ffa497fa640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:30:14.740 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.739+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa30004eb0 con 0x7ffa4c0b1730 2024-03-25T18:30:14.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.749+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218833097 0 0) 0x7ffa4c137750 con 0x7ffa4c14be90 2024-03-25T18:30:14.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.749+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4c1368f0 con 0x7ffa4c14be90 2024-03-25T18:30:14.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.749+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ffa44001b00 con 0x7ffa4c14be90 2024-03-25T18:30:14.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.749+0000 7ffa497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:14.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.749+0000 7ffa497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:14.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.749+0000 7ffa497fa640 10 monclient(hunting): dump: 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:14.750 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 901238157 0 0) 0x7ffa30004eb0 con 0x7ffa4c0b1730 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4c137750 con 0x7ffa4c0b1730 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ffa40001bc0 con 0x7ffa4c0b1730 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 10 monclient(hunting): dump: 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:14.756 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3334079663 0 0) 0x7ffa4c1363d0 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 >> v1:172.21.15.42:6790/0 conn(0x7ffa4c14be90 legacy=0x7ffa4c1d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 >> v1:172.21.15.42:6789/0 conn(0x7ffa4c0b1730 legacy=0x7ffa4c135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 monclient: found mon.b 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:14.757 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c130470 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 10 monclient: _renew_subs 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c130720 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 10 monclient: _renew_subs 2024-03-25T18:30:14.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:14.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c1322f0 con 0x7ffa4c0b4d60 2024-03-25T18:30:14.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.755+0000 7ffa52d3d640 1 librados: init done 2024-03-25T18:30:14.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.756+0000 7ffa52d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:14.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.756+0000 7ffa52d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:14.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.756+0000 7ffa52d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:14.760 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:14.756+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c136210 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa380047e0 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:15.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ffa38004c30 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.084 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa497fa640 10 monclient: dump: 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:15.085 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa4affd640 4 set_mon_vals no callback set 2024-03-25T18:30:15.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.084+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ffa38004e50 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.085+0000 7ffa50ad9640 10 monclient: get_auth_request con 0x7ffa3006e160 auth_method 0 2024-03-25T18:30:15.086 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.085+0000 7ffa497fa640 10 client.4714.objecter ms_handle_connect 0x7ffa3006e160 2024-03-25T18:30:15.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.092+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7ffa380547f0 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.092+0000 7ffa497fa640 10 client.4714.objecter ms_dispatch 0x7ffa4c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:15.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.092+0000 7ffa497fa640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:15.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.092+0000 7ffa497fa640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:15.092 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.092+0000 7ffa497fa640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T18:30:15.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.093+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa38050300 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.093+0000 7ffa497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:15.093 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.093+0000 7ffa497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:15.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.349+0000 7ffa52d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:15.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.349+0000 7ffa52d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:15.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.349+0000 7ffa52d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:15.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.349+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4c136560 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.378 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.376+0000 7ffa497fa640 1 -- 172.21.15.42:0/2737516455 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7ffa38052230 con 0x7ffa4c0b4d60 2024-03-25T18:30:15.378 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.376+0000 7ffa497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:15.378 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.376+0000 7ffa497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:15.378 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:15.378 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:15.380 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 10 librados: watch_flush enter 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 10 librados: watch_flush exit 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 10 client.4714.objecter successfully canceled tick 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 >> v1:172.21.15.42:6800/3081750888 conn(0x7ffa3006e160 legacy=0x7ffa30070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 10 monclient: shutdown 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 >> v1:172.21.15.120:6789/0 conn(0x7ffa4c0b4d60 legacy=0x7ffa4c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 shutdown_connections 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 >> 172.21.15.42:0/2737516455 conn(0x7ffa4c0617a0 msgr2=0x7ffa4c138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 shutdown_connections 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 1 -- 172.21.15.42:0/2737516455 wait complete. 2024-03-25T18:30:15.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:15.379+0000 7ffa52d3d640 1 librados: shutdown 2024-03-25T18:30:15.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:22.200 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.313+0000 7ff084cae640 10 monclient: get_monmap_and_config 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.313+0000 7ff084cae640 10 monclient: build_initial_monmap 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.313+0000 7ff084cae640 10 monclient: monmap: 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:30:22.314837+0000 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:30:22.314837+0000 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:22.315 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:22.316 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:22.316 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:30:22.316 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:30:22.316 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:22.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.314+0000 7ff084cae640 1 Processor -- start 2024-03-25T18:30:22.317 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.314+0000 7ff084cae640 1 -- start start 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.314+0000 7ff084cae640 10 monclient: init 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 10 monclient(hunting): picked mon.noname-b con 0x7ff0800b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 10 monclient(hunting): picked mon.noname-a con 0x7ff0800b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 10 monclient(hunting): picked mon.noname-c con 0x7ff08014be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0800af940 con 0x7ff0800b1730 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0800b0a70 con 0x7ff08014be90 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0800b0d50 con 0x7ff0800b4d60 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:22.318 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.315+0000 7ff084cae640 10 monclient(hunting): authenticate will time out at 7937.542482s 2024-03-25T18:30:22.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.329+0000 7ff07ed9d640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff08014be90 0x7ff08014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:51048/0 (socket says 172.21.15.42:51048) 2024-03-25T18:30:22.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.329+0000 7ff07ed9d640 1 -- 172.21.15.42:0/3293004156 learned_addr learned my addr 172.21.15.42:0/3293004156 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:22.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.346+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386647093 0 0) 0x7ff0800b0d50 con 0x7ff0800b4d60 2024-03-25T18:30:22.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.346+0000 7ff07d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:22.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.346+0000 7ff07d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:22.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.346+0000 7ff07d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:22.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.346+0000 7ff07d59a640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:30:22.347 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.346+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff068003520 con 0x7ff0800b4d60 2024-03-25T18:30:22.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921396414 0 0) 0x7ff0800af940 con 0x7ff0800b1730 2024-03-25T18:30:22.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:22.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:22.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:22.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:30:22.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0800b0d50 con 0x7ff0800b1730 2024-03-25T18:30:22.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007339077 0 0) 0x7ff0800b0a70 con 0x7ff08014be90 2024-03-25T18:30:22.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:30:22.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:22.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:22.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:30:22.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.348+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0800af940 con 0x7ff08014be90 2024-03-25T18:30:22.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952293494 0 0) 0x7ff0800af940 con 0x7ff08014be90 2024-03-25T18:30:22.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 >> v1:172.21.15.42:6790/0 conn(0x7ff0800b4d60 legacy=0x7ff08013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:22.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 >> v1:172.21.15.42:6789/0 conn(0x7ff0800b1730 legacy=0x7ff0800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:22.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 1 monclient: found mon.noname-c 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0800b1030 con 0x7ff08014be90 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 10 monclient: _finish_auth 0 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff078001930 con 0x7ff08014be90 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff07d59a640 10 monclient: dump: 2024-03-25T18:30:22.363 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff084cae640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:30:22.364 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.361+0000 7ff084cae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:22.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff078001a90 con 0x7ff08014be90 2024-03-25T18:30:22.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff07d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:22.381 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff07d59a640 1 -- 172.21.15.42:0/3293004156 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff078001ee0 con 0x7ff08014be90 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff07d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff07d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff07d59a640 10 monclient: dump: 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:22.382 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:22.383 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:22.383 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:22.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff084cae640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:22.383 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.380+0000 7ff084cae640 4 set_mon_vals no callback set 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.381+0000 7ff084cae640 10 monclient: shutdown 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.381+0000 7ff084cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.381+0000 7ff084cae640 1 -- 172.21.15.42:0/3293004156 >> v1:172.21.15.120:6789/0 conn(0x7ff08014be90 legacy=0x7ff08014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.381+0000 7ff084cae640 1 -- 172.21.15.42:0/3293004156 shutdown_connections 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.381+0000 7ff084cae640 1 -- 172.21.15.42:0/3293004156 >> 172.21.15.42:0/3293004156 conn(0x7ff0800617a0 msgr2=0x7ff080061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.381+0000 7ff084cae640 1 -- 172.21.15.42:0/3293004156 shutdown_connections 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.381+0000 7ff084cae640 1 -- 172.21.15.42:0/3293004156 wait complete. 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.382+0000 7ff084cae640 10 monclient: build_initial_monmap 2024-03-25T18:30:22.385 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.382+0000 7ff084cae640 10 monclient: monmap: 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.382+0000 7ff084cae640 1 librados: starting msgr at 2024-03-25T18:30:22.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.382+0000 7ff084cae640 1 librados: starting objecter 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.382+0000 7ff084cae640 1 Processor -- start 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 1 -- start start 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 1 librados: setting wanted keys 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 1 librados: calling monclient init 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient: init 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient(hunting): picked mon.noname-a con 0x7ff0800b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient(hunting): picked mon.noname-b con 0x7ff0800b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient(hunting): picked mon.noname-c con 0x7ff08014be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0801dac00 con 0x7ff0800b1730 2024-03-25T18:30:22.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff080134920 con 0x7ff0800b4d60 2024-03-25T18:30:22.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff080134ab0 con 0x7ff08014be90 2024-03-25T18:30:22.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:22.388 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.383+0000 7ff084cae640 10 monclient(hunting): authenticate will time out at 7937.610512s 2024-03-25T18:30:22.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.390+0000 7ff07e59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff0800b1730 0x7ff080132eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55750/0 (socket says 172.21.15.42:55750) 2024-03-25T18:30:22.391 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.390+0000 7ff07e59c640 1 -- 172.21.15.42:0/3758808345 learned_addr learned my addr 172.21.15.42:0/3758808345 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:22.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.391+0000 7ff06effd640 10 client.?.objecter ms_handle_connect 0x7ff0800b1730 2024-03-25T18:30:22.392 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.391+0000 7ff06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:22.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.395+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921825914 0 0) 0x7ff0801dac00 con 0x7ff0800b1730 2024-03-25T18:30:22.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.395+0000 7ff06effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:22.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.395+0000 7ff06effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:22.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.395+0000 7ff06effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:22.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.395+0000 7ff06effd640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:30:22.397 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.395+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff060004eb0 con 0x7ff0800b1730 2024-03-25T18:30:22.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.401+0000 7ff06effd640 10 client.?.objecter ms_handle_connect 0x7ff0800b4d60 2024-03-25T18:30:22.402 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.401+0000 7ff06effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.402+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3349883505 0 0) 0x7ff060004eb0 con 0x7ff0800b1730 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.402+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0801dac00 con 0x7ff0800b1730 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.402+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff0700019b0 con 0x7ff0800b1730 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.402+0000 7ff06effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.402+0000 7ff06effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.402+0000 7ff06effd640 10 monclient(hunting): dump: 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:22.404 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:22.405 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.402+0000 7ff06effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:22.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1881164788 0 0) 0x7ff0801dac00 con 0x7ff0800b1730 2024-03-25T18:30:22.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 >> v1:172.21.15.42:6790/0 conn(0x7ff08014be90 legacy=0x7ff080139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:22.412 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 >> v1:172.21.15.120:6789/0 conn(0x7ff0800b4d60 legacy=0x7ff0801db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 1 monclient: found mon.a 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0801335c0 con 0x7ff0800b1730 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 10 monclient: _finish_auth 0 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff06effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 5 monclient: authenticate success, global_id 4707 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 10 monclient: _renew_subs 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:30:22.413 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0801366d0 con 0x7ff0800b1730 2024-03-25T18:30:22.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:22.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 10 monclient: _renew_subs 2024-03-25T18:30:22.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:30:22.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0801db980 con 0x7ff0800b1730 2024-03-25T18:30:22.414 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.411+0000 7ff084cae640 1 librados: init done 2024-03-25T18:30:22.415 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.412+0000 7ff084cae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:22.415 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.412+0000 7ff084cae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:22.415 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.412+0000 7ff084cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:30:22.415 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.412+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff080133810 con 0x7ff0800b1730 2024-03-25T18:30:22.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.737+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff070001c90 con 0x7ff0800b1730 2024-03-25T18:30:22.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.737+0000 7ff06effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:22.738 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.737+0000 7ff07cd99640 4 set_mon_vals no callback set 2024-03-25T18:30:22.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.747+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff070003a60 con 0x7ff0800b1730 2024-03-25T18:30:22.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.747+0000 7ff06effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:22.748 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.747+0000 7ff06effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.747+0000 7ff06effd640 10 monclient: dump: 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:22.749 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:22.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.748+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff070003c40 con 0x7ff0800b1730 2024-03-25T18:30:22.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.749+0000 7ff07dd9b640 10 monclient: get_auth_request con 0x7ff06006db90 auth_method 0 2024-03-25T18:30:22.750 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.749+0000 7ff06effd640 10 client.4707.objecter ms_handle_connect 0x7ff06006db90 2024-03-25T18:30:22.758 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7ff0700548b0 con 0x7ff0800b1730 2024-03-25T18:30:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 10 client.4707.objecter ms_dispatch 0x7ff080000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T18:30:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0700503c0 con 0x7ff0800b1730 2024-03-25T18:30:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:22.759 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:22.757+0000 7ff06effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:23.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.009+0000 7ff084cae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:23.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.009+0000 7ff084cae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:23.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.009+0000 7ff084cae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:30:23.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.009+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0800627b0 con 0x7ff0800b1730 2024-03-25T18:30:23.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.072+0000 7ff06effd640 1 -- 172.21.15.42:0/3758808345 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7ff0700522f0 con 0x7ff0800b1730 2024-03-25T18:30:23.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.072+0000 7ff06effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:23.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.072+0000 7ff06effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:23.074 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:23.074 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:23.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.074+0000 7ff084cae640 10 librados: watch_flush enter 2024-03-25T18:30:23.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 10 librados: watch_flush exit 2024-03-25T18:30:23.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 10 client.4707.objecter successfully canceled tick 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff06006db90 legacy=0x7ff060070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 10 monclient: shutdown 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 >> v1:172.21.15.42:6789/0 conn(0x7ff0800b1730 legacy=0x7ff080132eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 shutdown_connections 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 >> 172.21.15.42:0/3758808345 conn(0x7ff0800617a0 msgr2=0x7ff08013c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 shutdown_connections 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 1 -- 172.21.15.42:0/3758808345 wait complete. 2024-03-25T18:30:23.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:23.075+0000 7ff084cae640 1 librados: shutdown 2024-03-25T18:30:23.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:29.896 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:30.010 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.009+0000 7ff93db82640 10 monclient: get_monmap_and_config 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.009+0000 7ff93db82640 10 monclient: build_initial_monmap 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.009+0000 7ff93db82640 10 monclient: monmap: 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:30:30.010665+0000 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:30:30.010665+0000 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:30:30.011 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.010+0000 7ff93db82640 1 Processor -- start 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.010+0000 7ff93db82640 1 -- start start 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.010+0000 7ff93db82640 10 monclient: init 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 10 monclient(hunting): picked mon.noname-b con 0x7ff9380a98f0 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 10 monclient(hunting): picked mon.noname-c con 0x7ff93813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:30.013 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 10 monclient(hunting): picked mon.noname-a con 0x7ff9381430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:30.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff938142520 con 0x7ff9381430c0 2024-03-25T18:30:30.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380b28a0 con 0x7ff93813f710 2024-03-25T18:30:30.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380b2a30 con 0x7ff9380a98f0 2024-03-25T18:30:30.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:30.014 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.011+0000 7ff93db82640 10 monclient(hunting): authenticate will time out at 7945.238290s 2024-03-25T18:30:30.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.023+0000 7ff936ffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff93813f710 0x7ff9381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:59254/0 (socket says 172.21.15.42:59254) 2024-03-25T18:30:30.024 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.023+0000 7ff936ffd640 1 -- 172.21.15.42:0/3236733875 learned_addr learned my addr 172.21.15.42:0/3236733875 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:30.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.035+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707591582 0 0) 0x7ff9380b28a0 con 0x7ff93813f710 2024-03-25T18:30:30.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.036+0000 7ff9367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:30.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.036+0000 7ff9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:30.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.036+0000 7ff9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:30.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.036+0000 7ff9367fc640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:30:30.037 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.036+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff920003520 con 0x7ff93813f710 2024-03-25T18:30:30.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.043+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607723554 0 0) 0x7ff9380b2a30 con 0x7ff9380a98f0 2024-03-25T18:30:30.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.043+0000 7ff9367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:30.044 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.043+0000 7ff9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:30.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.043+0000 7ff9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:30.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.043+0000 7ff9367fc640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:30:30.045 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.043+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9380b28a0 con 0x7ff9380a98f0 2024-03-25T18:30:30.046 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.044+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312194991 0 0) 0x7ff938142520 con 0x7ff9381430c0 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.044+0000 7ff9367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.044+0000 7ff9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.044+0000 7ff9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.044+0000 7ff9367fc640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.044+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9380b2a30 con 0x7ff9381430c0 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892611136 0 0) 0x7ff920003520 con 0x7ff93813f710 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 >> v1:172.21.15.42:6790/0 conn(0x7ff9380a98f0 legacy=0x7ff9380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 >> v1:172.21.15.42:6789/0 conn(0x7ff9381430c0 legacy=0x7ff93814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 1 monclient: found mon.noname-c 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:30:30.047 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9380b44c0 con 0x7ff93813f710 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff92c001690 con 0x7ff93813f710 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff9367fc640 10 monclient: dump: 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:30.048 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:30.049 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:30.049 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:30.049 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff93db82640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:30:30.049 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.045+0000 7ff93db82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff92c0017f0 con 0x7ff93813f710 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff9367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff9367fc640 1 -- 172.21.15.42:0/3236733875 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff92c001c00 con 0x7ff93813f710 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff9367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff9367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff9367fc640 10 monclient: dump: 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.072 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:30.073 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:30.073 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:30.073 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:30.073 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:30.073 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff93db82640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:30.073 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff93db82640 4 set_mon_vals no callback set 2024-03-25T18:30:30.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff93db82640 10 monclient: shutdown 2024-03-25T18:30:30.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff93db82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:30.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.071+0000 7ff93db82640 1 -- 172.21.15.42:0/3236733875 >> v1:172.21.15.120:6789/0 conn(0x7ff93813f710 legacy=0x7ff9381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:30.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.072+0000 7ff93db82640 1 -- 172.21.15.42:0/3236733875 shutdown_connections 2024-03-25T18:30:30.074 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.072+0000 7ff93db82640 1 -- 172.21.15.42:0/3236733875 >> 172.21.15.42:0/3236733875 conn(0x7ff9380b7ea0 msgr2=0x7ff9380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.072+0000 7ff93db82640 1 -- 172.21.15.42:0/3236733875 shutdown_connections 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.072+0000 7ff93db82640 1 -- 172.21.15.42:0/3236733875 wait complete. 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.072+0000 7ff93db82640 10 monclient: build_initial_monmap 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.072+0000 7ff93db82640 10 monclient: monmap: 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:30:30.075 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.073+0000 7ff93db82640 1 librados: starting msgr at 2024-03-25T18:30:30.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.073+0000 7ff93db82640 1 librados: starting objecter 2024-03-25T18:30:30.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.073+0000 7ff93db82640 1 Processor -- start 2024-03-25T18:30:30.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.073+0000 7ff93db82640 1 -- start start 2024-03-25T18:30:30.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.073+0000 7ff93db82640 1 librados: setting wanted keys 2024-03-25T18:30:30.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.073+0000 7ff93db82640 1 librados: calling monclient init 2024-03-25T18:30:30.076 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.073+0000 7ff93db82640 10 monclient: init 2024-03-25T18:30:30.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:30.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:30.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 10 monclient(hunting): picked mon.noname-a con 0x7ff9380a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:30.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 10 monclient(hunting): picked mon.noname-b con 0x7ff93813f710 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:30.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 10 monclient(hunting): picked mon.noname-c con 0x7ff9381430c0 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:30.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9381374f0 con 0x7ff9380a98f0 2024-03-25T18:30:30.077 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff938136a50 con 0x7ff93813f710 2024-03-25T18:30:30.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff938136be0 con 0x7ff9381430c0 2024-03-25T18:30:30.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:30.078 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.074+0000 7ff93db82640 10 monclient(hunting): authenticate will time out at 7945.300744s 2024-03-25T18:30:30.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.079+0000 7ff936ffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7ff93813f710 0x7ff938132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:59298/0 (socket says 172.21.15.42:59298) 2024-03-25T18:30:30.080 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.079+0000 7ff936ffd640 1 -- 172.21.15.42:0/3031580203 learned_addr learned my addr 172.21.15.42:0/3031580203 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:30.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.080+0000 7ff91ffff640 10 client.?.objecter ms_handle_connect 0x7ff93813f710 2024-03-25T18:30:30.081 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.080+0000 7ff91ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:30.097 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.096+0000 7ff91ffff640 10 client.?.objecter ms_handle_connect 0x7ff9380a98f0 2024-03-25T18:30:30.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.096+0000 7ff91ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:30.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.097+0000 7ff91ffff640 10 client.?.objecter ms_handle_connect 0x7ff9381430c0 2024-03-25T18:30:30.098 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.097+0000 7ff91ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:30.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.100+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364143626 0 0) 0x7ff938136a50 con 0x7ff93813f710 2024-03-25T18:30:30.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.100+0000 7ff91ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:30.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.100+0000 7ff91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:30.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.100+0000 7ff91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:30.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.100+0000 7ff91ffff640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:30:30.101 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.100+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff924004eb0 con 0x7ff93813f710 2024-03-25T18:30:30.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.103+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38335286 0 0) 0x7ff938136be0 con 0x7ff9381430c0 2024-03-25T18:30:30.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.103+0000 7ff91ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:30.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.103+0000 7ff91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:30.104 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.103+0000 7ff91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:30.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.103+0000 7ff91ffff640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:30:30.105 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.103+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff938136a50 con 0x7ff9381430c0 2024-03-25T18:30:30.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.104+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867049957 0 0) 0x7ff9381374f0 con 0x7ff9380a98f0 2024-03-25T18:30:30.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.105+0000 7ff91ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:30.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.105+0000 7ff91ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:30.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.105+0000 7ff91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:30.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.105+0000 7ff91ffff640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:30:30.106 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.105+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff938136be0 con 0x7ff9380a98f0 2024-03-25T18:30:30.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.113+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4171582039 0 0) 0x7ff938136a50 con 0x7ff9381430c0 2024-03-25T18:30:30.114 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.113+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9381374f0 con 0x7ff9381430c0 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.113+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff930001b50 con 0x7ff9381430c0 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.113+0000 7ff91ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.113+0000 7ff91ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.113+0000 7ff91ffff640 10 monclient(hunting): dump: 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:30.115 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.116 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.113+0000 7ff91ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.118+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1372430632 0 0) 0x7ff924004eb0 con 0x7ff93813f710 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.118+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff938136a50 con 0x7ff93813f710 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.118+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff92c001bc0 con 0x7ff93813f710 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.118+0000 7ff91ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.118+0000 7ff91ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.118+0000 7ff91ffff640 10 monclient(hunting): dump: 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:30.120 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:30.121 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:30.121 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.129+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2161509662 0 0) 0x7ff938136be0 con 0x7ff9380a98f0 2024-03-25T18:30:30.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.129+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff924004eb0 con 0x7ff9380a98f0 2024-03-25T18:30:30.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.129+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff9280018f0 con 0x7ff9380a98f0 2024-03-25T18:30:30.130 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.129+0000 7ff91ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.129+0000 7ff91ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.129+0000 7ff91ffff640 10 monclient(hunting): dump: 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:30.131 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656003541 0 0) 0x7ff9381374f0 con 0x7ff9381430c0 2024-03-25T18:30:30.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 >> v1:172.21.15.120:6789/0 conn(0x7ff93813f710 legacy=0x7ff938132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:30.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 >> v1:172.21.15.42:6789/0 conn(0x7ff9380a98f0 legacy=0x7ff9381dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:30.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:30.141 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 1 monclient: found mon.c 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938136d70 con 0x7ff9381430c0 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 10 monclient: _finish_auth 0 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff91ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 10 monclient: _renew_subs 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9380b71c0 con 0x7ff9381430c0 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 10 client.4718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 10 monclient: _renew_subs 2024-03-25T18:30:30.142 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:30:30.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff938133680 con 0x7ff9381430c0 2024-03-25T18:30:30.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.139+0000 7ff93db82640 1 librados: init done 2024-03-25T18:30:30.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.140+0000 7ff93db82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:30.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.140+0000 7ff93db82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:30.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.140+0000 7ff93db82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:30:30.143 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.140+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9380627b0 con 0x7ff9381430c0 2024-03-25T18:30:30.451 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.450+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff930001fe0 con 0x7ff9381430c0 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.450+0000 7ff91ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.450+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7ff930003b30 con 0x7ff9381430c0 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.450+0000 7ff91ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.450+0000 7ff91ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.450+0000 7ff91ffff640 10 monclient: dump: 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:30.452 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:30.453 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:30.453 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:30.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.450+0000 7ff935ffb640 4 set_mon_vals no callback set 2024-03-25T18:30:30.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.451+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7ff930019b00 con 0x7ff9381430c0 2024-03-25T18:30:30.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.451+0000 7ff9377fe640 10 monclient: get_auth_request con 0x7ff92406e0b0 auth_method 0 2024-03-25T18:30:30.453 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.452+0000 7ff91ffff640 10 client.4718.objecter ms_handle_connect 0x7ff92406e0b0 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7ff930053880 con 0x7ff9381430c0 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 10 client.4718.objecter ms_dispatch 0x7ff938000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 3 client.4718.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 3 client.4718.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 20 client.4718.objecter dump_active .. 0 homeless 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff93004f1c0 con 0x7ff9381430c0 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:30.460 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.458+0000 7ff91ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:30.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.679+0000 7ff93db82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:30.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.679+0000 7ff93db82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:30.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.679+0000 7ff93db82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.42:6790/0 2024-03-25T18:30:30.680 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.679+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 --> v1:172.21.15.42:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff93813e2b0 con 0x7ff9381430c0 2024-03-25T18:30:30.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.712+0000 7ff91ffff640 1 -- 172.21.15.42:0/3031580203 <== mon.2 v1:172.21.15.42:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7ff9300512c0 con 0x7ff9381430c0 2024-03-25T18:30:30.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.712+0000 7ff91ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:30.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.712+0000 7ff91ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:30.714 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:30.714 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:30.717 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.715+0000 7ff93db82640 10 librados: watch_flush enter 2024-03-25T18:30:30.717 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.715+0000 7ff93db82640 10 librados: watch_flush exit 2024-03-25T18:30:30.717 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.715+0000 7ff93db82640 20 client.4718.objecter shutdown clearing up homeless session... 2024-03-25T18:30:30.717 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.715+0000 7ff93db82640 10 client.4718.objecter successfully canceled tick 2024-03-25T18:30:30.717 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.715+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 >> v1:172.21.15.42:6800/3081750888 conn(0x7ff92406e0b0 legacy=0x7ff924070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 10 monclient: shutdown 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 >> v1:172.21.15.42:6790/0 conn(0x7ff9381430c0 legacy=0x7ff9381e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 shutdown_connections 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 >> 172.21.15.42:0/3031580203 conn(0x7ff9380b7ea0 msgr2=0x7ff938074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 shutdown_connections 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 1 -- 172.21.15.42:0/3031580203 wait complete. 2024-03-25T18:30:30.718 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:30.716+0000 7ff93db82640 1 librados: shutdown 2024-03-25T18:30:30.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:33.599 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:16.768 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.649 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:16.818 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.650 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:16.770 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.690 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:16.859 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.706 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:16.827 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.713 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:16.882 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.724 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:16.893 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.728 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:16.848 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.756 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:16.926 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:37.536 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:37.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.648+0000 7f324820e640 10 monclient: get_monmap_and_config 2024-03-25T18:30:37.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.648+0000 7f324820e640 10 monclient: build_initial_monmap 2024-03-25T18:30:37.649 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.648+0000 7f324820e640 10 monclient: monmap: 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:30:37.649609+0000 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:30:37.649609+0000 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:30:37.650 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.649+0000 7f324820e640 1 Processor -- start 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.649+0000 7f324820e640 1 -- start start 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.649+0000 7f324820e640 10 monclient: init 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 10 monclient(hunting): picked mon.noname-c con 0x7f32400a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 10 monclient(hunting): picked mon.noname-b con 0x7f324013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:37.651 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 10 monclient(hunting): picked mon.noname-a con 0x7f32401430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:37.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3240142520 con 0x7f32401430c0 2024-03-25T18:30:37.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32400b28a0 con 0x7f32400a98f0 2024-03-25T18:30:37.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32400b2a30 con 0x7f324013f710 2024-03-25T18:30:37.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:37.652 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.650+0000 7f324820e640 10 monclient(hunting): authenticate will time out at 7952.876898s 2024-03-25T18:30:37.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.658+0000 7f32457a9640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f324013f710 0x7f32401429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:43520/0 (socket says 172.21.15.42:43520) 2024-03-25T18:30:37.659 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.658+0000 7f32457a9640 1 -- 172.21.15.42:0/3403936265 learned_addr learned my addr 172.21.15.42:0/3403936265 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:37.670 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.669+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740874376 0 0) 0x7f32400b2a30 con 0x7f324013f710 2024-03-25T18:30:37.670 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.669+0000 7f3244fa8640 10 monclient(hunting): _negotiate 2024-03-25T18:30:37.670 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.669+0000 7f3244fa8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:37.670 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.669+0000 7f3244fa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:37.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.669+0000 7f3244fa8640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:30:37.671 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.669+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f322c003520 con 0x7f324013f710 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2876784612 0 0) 0x7f322c003520 con 0x7f324013f710 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 >> v1:172.21.15.120:6789/0 conn(0x7f32400a98f0 legacy=0x7f32400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 >> v1:172.21.15.42:6789/0 conn(0x7f32401430c0 legacy=0x7f324014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 10 monclient: _finish_hunting 0 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 1 monclient: found mon.noname-b 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.42:6790/0 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 --> v1:172.21.15.42:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32400b44c0 con 0x7f324013f710 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 10 monclient: _finish_auth 0 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 10 monclient: _check_auth_tickets 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3230001690 con 0x7f324013f710 2024-03-25T18:30:37.677 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f324820e640 5 monclient: authenticate success, global_id 4721 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f324820e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f3244fa8640 10 monclient: dump: 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:37.678 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:37.679 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:37.679 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.675+0000 7f324820e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:37.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 <== mon.2 v1:172.21.15.42:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3230001860 con 0x7f324013f710 2024-03-25T18:30:37.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f3244fa8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:37.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f3244fa8640 1 -- 172.21.15.42:0/3403936265 <== mon.2 v1:172.21.15.42:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3230001cb0 con 0x7f324013f710 2024-03-25T18:30:37.701 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f324820e640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f324820e640 4 set_mon_vals no callback set 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f3244fa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f3244fa8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.700+0000 7f3244fa8640 10 monclient: dump: 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:37.702 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:37.703 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.701+0000 7f324820e640 10 monclient: shutdown 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.701+0000 7f324820e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.701+0000 7f324820e640 1 -- 172.21.15.42:0/3403936265 >> v1:172.21.15.42:6790/0 conn(0x7f324013f710 legacy=0x7f32401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.701+0000 7f324820e640 1 -- 172.21.15.42:0/3403936265 shutdown_connections 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.701+0000 7f324820e640 1 -- 172.21.15.42:0/3403936265 >> 172.21.15.42:0/3403936265 conn(0x7f32400b7ea0 msgr2=0x7f32400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.701+0000 7f324820e640 1 -- 172.21.15.42:0/3403936265 shutdown_connections 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.701+0000 7f324820e640 1 -- 172.21.15.42:0/3403936265 wait complete. 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 10 monclient: build_initial_monmap 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 10 monclient: monmap: 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:37.704 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 1 librados: starting msgr at 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 1 librados: starting objecter 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 1 Processor -- start 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 1 -- start start 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 1 librados: setting wanted keys 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 1 librados: calling monclient init 2024-03-25T18:30:37.705 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.702+0000 7f324820e640 10 monclient: init 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 10 monclient(hunting): picked mon.noname-a con 0x7f32400a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 10 monclient(hunting): picked mon.noname-c con 0x7f324013f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 10 monclient(hunting): picked mon.noname-b con 0x7f32401430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32400b7890 con 0x7f32400a98f0 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324012d970 con 0x7f32401430c0 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324013ba50 con 0x7f324013f710 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:37.707 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.703+0000 7f324820e640 10 monclient(hunting): authenticate will time out at 7952.930443s 2024-03-25T18:30:37.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.713+0000 7f32457a9640 1 --1- >> v1:172.21.15.42:6790/0 conn(0x7f324013f710 0x7f324012cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6790/0 says I am v1:172.21.15.42:43552/0 (socket says 172.21.15.42:43552) 2024-03-25T18:30:37.714 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.713+0000 7f32457a9640 1 -- 172.21.15.42:0/2104248452 learned_addr learned my addr 172.21.15.42:0/2104248452 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:37.720 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.719+0000 7f32367fc640 10 client.?.objecter ms_handle_connect 0x7f324013f710 2024-03-25T18:30:37.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.719+0000 7f32367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:37.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.720+0000 7f32367fc640 10 client.?.objecter ms_handle_connect 0x7f32400a98f0 2024-03-25T18:30:37.721 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.720+0000 7f32367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:37.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.723+0000 7f32367fc640 10 client.?.objecter ms_handle_connect 0x7f32401430c0 2024-03-25T18:30:37.724 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.723+0000 7f32367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:37.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.735+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585234307 0 0) 0x7f324012d970 con 0x7f32401430c0 2024-03-25T18:30:37.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.735+0000 7f32367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:37.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.735+0000 7f32367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:37.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.735+0000 7f32367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:37.736 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.735+0000 7f32367fc640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:30:37.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.735+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3224004eb0 con 0x7f32401430c0 2024-03-25T18:30:37.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.736+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537080178 0 0) 0x7f324013ba50 con 0x7f324013f710 2024-03-25T18:30:37.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.736+0000 7f32367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:37.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.736+0000 7f32367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:37.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.736+0000 7f32367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:37.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.736+0000 7f32367fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:30:37.737 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.736+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f324012d970 con 0x7f324013f710 2024-03-25T18:30:37.743 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.742+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116034505 0 0) 0x7f32400b7890 con 0x7f32400a98f0 2024-03-25T18:30:37.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.742+0000 7f32367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:37.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.742+0000 7f32367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:37.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.742+0000 7f32367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:37.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.742+0000 7f32367fc640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:30:37.744 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.742+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f324013ba50 con 0x7f32400a98f0 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.750+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309341002 0 0) 0x7f3224004eb0 con 0x7f32401430c0 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.750+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32400b7890 con 0x7f32401430c0 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.750+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3238001ad0 con 0x7f32401430c0 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.750+0000 7f32367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.750+0000 7f32367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.750+0000 7f32367fc640 10 monclient(hunting): dump: 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:37.752 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:37.753 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:37.753 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:37.753 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:37.753 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:37.753 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.750+0000 7f32367fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:37.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.763+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778994579 0 0) 0x7f324013ba50 con 0x7f32400a98f0 2024-03-25T18:30:37.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.763+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3224004eb0 con 0x7f32400a98f0 2024-03-25T18:30:37.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.763+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f323c001830 con 0x7f32400a98f0 2024-03-25T18:30:37.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.763+0000 7f32367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:37.764 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.763+0000 7f32367fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.763+0000 7f32367fc640 10 monclient(hunting): dump: 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:37.765 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.764+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1555998414 0 0) 0x7f324012d970 con 0x7f324013f710 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.764+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f324013ba50 con 0x7f324013f710 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.764+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3230001b70 con 0x7f324013f710 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.764+0000 7f32367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.764+0000 7f32367fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.764+0000 7f32367fc640 10 monclient(hunting): dump: 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:37.766 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.767 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:37.767 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:37.767 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:37.767 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:37.767 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:37.767 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:37.767 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:37.768 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4278467001 0 0) 0x7f32400b7890 con 0x7f32401430c0 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 >> v1:172.21.15.42:6790/0 conn(0x7f324013f710 legacy=0x7f324012cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 >> v1:172.21.15.42:6789/0 conn(0x7f32400a98f0 legacy=0x7f324013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 1 monclient: found mon.b 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324013bd30 con 0x7f32401430c0 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f32367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 5 monclient: authenticate success, global_id 4732 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 10 monclient: _renew_subs 2024-03-25T18:30:37.769 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32401399a0 con 0x7f32401430c0 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 10 monclient: _renew_subs 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f324012fe20 con 0x7f32401430c0 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.767+0000 7f324820e640 1 librados: init done 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.768+0000 7f324820e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:37.770 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.768+0000 7f324820e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:37.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.768+0000 7f324820e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:37.771 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:37.768+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324014f2d0 con 0x7f32401430c0 2024-03-25T18:30:38.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3238001fe0 con 0x7f32401430c0 2024-03-25T18:30:38.066 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f32367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f3238003b10 con 0x7f32401430c0 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f32367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f32367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f32367fc640 10 monclient: dump: 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:38.067 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:38.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f3237fff640 4 set_mon_vals no callback set 2024-03-25T18:30:38.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.065+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f3238019ae0 con 0x7f32401430c0 2024-03-25T18:30:38.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.066+0000 7f3245faa640 10 monclient: get_auth_request con 0x7f322406e010 auth_method 0 2024-03-25T18:30:38.068 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.066+0000 7f32367fc640 10 client.4732.objecter ms_handle_connect 0x7f322406e010 2024-03-25T18:30:38.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.069+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7f3238053830 con 0x7f32401430c0 2024-03-25T18:30:38.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.069+0000 7f32367fc640 10 client.4732.objecter ms_dispatch 0x7f3240000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:38.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.069+0000 7f32367fc640 3 client.4732.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:38.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.069+0000 7f32367fc640 3 client.4732.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:38.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.070+0000 7f32367fc640 20 client.4732.objecter dump_active .. 0 homeless 2024-03-25T18:30:38.071 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.070+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f323804f170 con 0x7f32401430c0 2024-03-25T18:30:38.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.070+0000 7f32367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:38.072 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.070+0000 7f32367fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:38.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.303+0000 7f324820e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:38.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.303+0000 7f324820e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:38.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.303+0000 7f324820e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:38.304 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.303+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32400627b0 con 0x7f32401430c0 2024-03-25T18:30:38.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.331+0000 7f32367fc640 1 -- 172.21.15.42:0/2104248452 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7f3238051270 con 0x7f32401430c0 2024-03-25T18:30:38.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.331+0000 7f32367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:38.333 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.331+0000 7f32367fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:38.333 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:38.333 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:38.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.333+0000 7f324820e640 10 librados: watch_flush enter 2024-03-25T18:30:38.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 10 librados: watch_flush exit 2024-03-25T18:30:38.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 20 client.4732.objecter shutdown clearing up homeless session... 2024-03-25T18:30:38.335 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 10 client.4732.objecter successfully canceled tick 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 >> v1:172.21.15.42:6800/3081750888 conn(0x7f322406e010 legacy=0x7f32240704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 10 monclient: shutdown 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 >> v1:172.21.15.120:6789/0 conn(0x7f32401430c0 legacy=0x7f324012d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 shutdown_connections 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 >> 172.21.15.42:0/2104248452 conn(0x7f32400b7ea0 msgr2=0x7f3240074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 shutdown_connections 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 1 -- 172.21.15.42:0/2104248452 wait complete. 2024-03-25T18:30:38.336 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:38.334+0000 7f324820e640 1 librados: shutdown 2024-03-25T18:30:38.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:41.372 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:24.541 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.377 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:24.546 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.383 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:24.503 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.384 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:24.554 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.386 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:24.506 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.387 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:24.556 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.399 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:24.568 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.404 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:24.524 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:41.452 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:24.622 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:45.154 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:45.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.264+0000 7f82726d9640 10 monclient: get_monmap_and_config 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.264+0000 7f82726d9640 10 monclient: build_initial_monmap 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.264+0000 7f82726d9640 10 monclient: monmap: 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:30:45.265730+0000 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:30:45.265730+0000 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:30:45.266 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:45.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.265+0000 7f82726d9640 1 Processor -- start 2024-03-25T18:30:45.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 1 -- start start 2024-03-25T18:30:45.267 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient: init 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient(hunting): picked mon.noname-b con 0x7f826c0b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient(hunting): picked mon.noname-a con 0x7f826c0b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient(hunting): picked mon.noname-c con 0x7f826c14be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0af940 con 0x7f826c0b1730 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0b0a70 con 0x7f826c14be90 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0b0d50 con 0x7f826c0b4d60 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:45.268 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.266+0000 7f82726d9640 10 monclient(hunting): authenticate will time out at 7960.493391s 2024-03-25T18:30:45.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.285+0000 7f8263fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f826c0b1730 0x7f826c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43530/0 (socket says 172.21.15.42:43530) 2024-03-25T18:30:45.287 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.285+0000 7f8263fff640 1 -- 172.21.15.42:0/3143877920 learned_addr learned my addr 172.21.15.42:0/3143877920 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:45.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643769348 0 0) 0x7f826c0b0d50 con 0x7f826c0b4d60 2024-03-25T18:30:45.307 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8258003520 con 0x7f826c0b4d60 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186399338 0 0) 0x7f826c0af940 con 0x7f826c0b1730 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:30:45.308 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.306+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c0b0d50 con 0x7f826c0b1730 2024-03-25T18:30:45.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.309+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999440245 0 0) 0x7f826c0b0a70 con 0x7f826c14be90 2024-03-25T18:30:45.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.309+0000 7f826b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:45.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.309+0000 7f826b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:45.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.309+0000 7f826b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:45.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.309+0000 7f826b7fe640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:30:45.310 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.309+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c0af940 con 0x7f826c14be90 2024-03-25T18:30:45.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1432283841 0 0) 0x7f826c0af940 con 0x7f826c14be90 2024-03-25T18:30:45.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 >> v1:172.21.15.42:6790/0 conn(0x7f826c0b4d60 legacy=0x7f826c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.330 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 >> v1:172.21.15.42:6789/0 conn(0x7f826c0b1730 legacy=0x7f826c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0b1030 con 0x7f826c14be90 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8264001930 con 0x7f826c14be90 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f826b7fe640 10 monclient: dump: 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:45.331 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f82726d9640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:30:45.332 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.329+0000 7f82726d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8264001a90 con 0x7f826c14be90 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f826b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f826b7fe640 1 -- 172.21.15.42:0/3143877920 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f8264001ee0 con 0x7f826c14be90 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 4 set_mon_vals no callback set 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f826b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f826b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f826b7fe640 10 monclient: dump: 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:45.346 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:45.347 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:45.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 10 monclient: shutdown 2024-03-25T18:30:45.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:45.348 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 1 -- 172.21.15.42:0/3143877920 >> v1:172.21.15.120:6789/0 conn(0x7f826c14be90 legacy=0x7f826c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 1 -- 172.21.15.42:0/3143877920 shutdown_connections 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 1 -- 172.21.15.42:0/3143877920 >> 172.21.15.42:0/3143877920 conn(0x7f826c0617a0 msgr2=0x7f826c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.345+0000 7f82726d9640 1 -- 172.21.15.42:0/3143877920 shutdown_connections 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.346+0000 7f82726d9640 1 -- 172.21.15.42:0/3143877920 wait complete. 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.346+0000 7f82726d9640 10 monclient: build_initial_monmap 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.346+0000 7f82726d9640 10 monclient: monmap: 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:30:45.349 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.347+0000 7f82726d9640 1 librados: starting msgr at 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.347+0000 7f82726d9640 1 librados: starting objecter 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.347+0000 7f82726d9640 1 Processor -- start 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.347+0000 7f82726d9640 1 -- start start 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.347+0000 7f82726d9640 1 librados: setting wanted keys 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.347+0000 7f82726d9640 1 librados: calling monclient init 2024-03-25T18:30:45.350 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.347+0000 7f82726d9640 10 monclient: init 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 10 monclient(hunting): picked mon.noname-b con 0x7f826c0b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 10 monclient(hunting): picked mon.noname-a con 0x7f826c0b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 10 monclient(hunting): picked mon.noname-c con 0x7f826c14be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c13b490 con 0x7f826c0b4d60 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c13a220 con 0x7f826c0b1730 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c139030 con 0x7f826c14be90 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:45.351 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.348+0000 7f82726d9640 10 monclient(hunting): authenticate will time out at 7960.574771s 2024-03-25T18:30:45.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.356+0000 7f826bfff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f826c0b1730 0x7f826c139b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:38604/0 (socket says 172.21.15.42:38604) 2024-03-25T18:30:45.357 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.356+0000 7f826bfff640 1 -- 172.21.15.42:0/1275102226 learned_addr learned my addr 172.21.15.42:0/1275102226 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:45.360 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.359+0000 7f82697fa640 10 client.?.objecter ms_handle_connect 0x7f826c0b1730 2024-03-25T18:30:45.360 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.359+0000 7f82697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:45.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.361+0000 7f82697fa640 10 client.?.objecter ms_handle_connect 0x7f826c0b4d60 2024-03-25T18:30:45.362 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.361+0000 7f82697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:45.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.366+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481595575 0 0) 0x7f826c13a220 con 0x7f826c0b1730 2024-03-25T18:30:45.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.366+0000 7f82697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:45.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.366+0000 7f82697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:45.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.366+0000 7f82697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:45.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.366+0000 7f82697fa640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:30:45.367 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.366+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c004eb0 con 0x7f826c0b1730 2024-03-25T18:30:45.379 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.378+0000 7f82697fa640 10 client.?.objecter ms_handle_connect 0x7f826c14be90 2024-03-25T18:30:45.379 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.378+0000 7f82697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:45.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.385+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1237725948 0 0) 0x7f824c004eb0 con 0x7f826c0b1730 2024-03-25T18:30:45.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.385+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f826c13a220 con 0x7f826c0b1730 2024-03-25T18:30:45.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.385+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f825c0019b0 con 0x7f826c0b1730 2024-03-25T18:30:45.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.385+0000 7f82697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:45.386 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.385+0000 7f82697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.385+0000 7f82697fa640 10 monclient(hunting): dump: 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:45.387 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.385+0000 7f82697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:45.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.389+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255220845 0 0) 0x7f826c13b490 con 0x7f826c0b4d60 2024-03-25T18:30:45.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.389+0000 7f82697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:45.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.389+0000 7f82697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:45.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.389+0000 7f82697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:45.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.389+0000 7f82697fa640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:30:45.390 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.389+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c004eb0 con 0x7f826c0b4d60 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3434007395 0 0) 0x7f826c13a220 con 0x7f826c0b1730 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 >> v1:172.21.15.42:6790/0 conn(0x7f826c14be90 legacy=0x7f826c1dae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 >> v1:172.21.15.42:6789/0 conn(0x7f826c0b4d60 legacy=0x7f826c132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 1 monclient: found mon.b 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c13bb20 con 0x7f826c0b1730 2024-03-25T18:30:45.394 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 10 monclient: _renew_subs 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c1334c0 con 0x7f826c0b1730 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 10 client.4738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 10 monclient: _renew_subs 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c1db710 con 0x7f826c0b1730 2024-03-25T18:30:45.395 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.392+0000 7f82726d9640 1 librados: init done 2024-03-25T18:30:45.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.394+0000 7f82726d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:45.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.394+0000 7f82726d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:45.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.394+0000 7f82726d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:45.396 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.394+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c1401f0 con 0x7f826c0b1730 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.659+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f825c001c50 con 0x7f826c0b1730 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.659+0000 7f82697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.659+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f825c003a60 con 0x7f826c0b1730 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.659+0000 7f82697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.659+0000 7f82697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.659+0000 7f82697fa640 10 monclient: dump: 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:45.661 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:45.662 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:45.662 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:45.662 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:45.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.659+0000 7f826affd640 4 set_mon_vals no callback set 2024-03-25T18:30:45.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.660+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f825c003c80 con 0x7f826c0b1730 2024-03-25T18:30:45.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.661+0000 7f8263fff640 10 monclient: get_auth_request con 0x7f824c06dd60 auth_method 0 2024-03-25T18:30:45.662 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.661+0000 7f82697fa640 10 client.4738.objecter ms_handle_connect 0x7f824c06dd60 2024-03-25T18:30:45.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.666+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7f825c054840 con 0x7f826c0b1730 2024-03-25T18:30:45.667 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.666+0000 7f82697fa640 10 client.4738.objecter ms_dispatch 0x7f826c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:45.668 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.666+0000 7f82697fa640 3 client.4738.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:45.668 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.666+0000 7f82697fa640 3 client.4738.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:45.668 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.666+0000 7f82697fa640 20 client.4738.objecter dump_active .. 0 homeless 2024-03-25T18:30:45.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.667+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f825c050350 con 0x7f826c0b1730 2024-03-25T18:30:45.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.667+0000 7f82697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:45.669 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.667+0000 7f82697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:45.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.921+0000 7f82726d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:45.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.921+0000 7f82726d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:45.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.921+0000 7f82726d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:45.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.921+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f826c1dba10 con 0x7f826c0b1730 2024-03-25T18:30:45.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.945+0000 7f82697fa640 1 -- 172.21.15.42:0/1275102226 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7f826c1dba10 con 0x7f826c0b1730 2024-03-25T18:30:45.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.945+0000 7f82697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:45.946 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.945+0000 7f82697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:45.946 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:45.947 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 10 librados: watch_flush enter 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 10 librados: watch_flush exit 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 20 client.4738.objecter shutdown clearing up homeless session... 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 10 client.4738.objecter successfully canceled tick 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 >> v1:172.21.15.42:6800/3081750888 conn(0x7f824c06dd60 legacy=0x7f824c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 10 monclient: shutdown 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 >> v1:172.21.15.120:6789/0 conn(0x7f826c0b1730 legacy=0x7f826c139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 shutdown_connections 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 >> 172.21.15.42:0/1275102226 conn(0x7f826c0617a0 msgr2=0x7f826c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 shutdown_connections 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 1 -- 172.21.15.42:0/1275102226 wait complete. 2024-03-25T18:30:45.949 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:45.947+0000 7f82726d9640 1 librados: shutdown 2024-03-25T18:30:45.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:52.766 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.879+0000 7fed29df6640 10 monclient: get_monmap_and_config 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.879+0000 7fed29df6640 10 monclient: build_initial_monmap 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.879+0000 7fed29df6640 10 monclient: monmap: 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:30:52.880968+0000 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:30:52.880968+0000 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:52.881 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:30:52.882 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:30:52.882 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:52.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.880+0000 7fed29df6640 1 Processor -- start 2024-03-25T18:30:52.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.881+0000 7fed29df6640 1 -- start start 2024-03-25T18:30:52.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.881+0000 7fed29df6640 10 monclient: init 2024-03-25T18:30:52.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.881+0000 7fed29df6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:52.883 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.881+0000 7fed29df6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.881+0000 7fed29df6640 10 monclient(hunting): picked mon.noname-b con 0x7fed240b4d60 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.881+0000 7fed29df6640 10 monclient(hunting): picked mon.noname-a con 0x7fed240b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.882+0000 7fed29df6640 10 monclient(hunting): picked mon.noname-c con 0x7fed2414be90 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.882+0000 7fed29df6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240af940 con 0x7fed240b1730 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.882+0000 7fed29df6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240b0a70 con 0x7fed2414be90 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.882+0000 7fed29df6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240b0d50 con 0x7fed240b4d60 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.882+0000 7fed29df6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:52.884 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.882+0000 7fed29df6640 10 monclient(hunting): authenticate will time out at 7968.109031s 2024-03-25T18:30:52.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.887+0000 7fed23fff640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fed2414be90 0x7fed2414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:46448/0 (socket says 172.21.15.42:46448) 2024-03-25T18:30:52.889 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.887+0000 7fed23fff640 1 -- 172.21.15.42:0/4268853962 learned_addr learned my addr 172.21.15.42:0/4268853962 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:52.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.901+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931014822 0 0) 0x7fed240b0a70 con 0x7fed2414be90 2024-03-25T18:30:52.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.902+0000 7fed227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:52.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.902+0000 7fed227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:52.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.902+0000 7fed227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:52.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.902+0000 7fed227fc640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:30:52.903 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.902+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0c003520 con 0x7fed2414be90 2024-03-25T18:30:52.912 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.911+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464126809 0 0) 0x7fed240af940 con 0x7fed240b1730 2024-03-25T18:30:52.912 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.911+0000 7fed227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:52.912 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.911+0000 7fed227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:52.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.911+0000 7fed227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:52.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.911+0000 7fed227fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:30:52.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.911+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed240b0a70 con 0x7fed240b1730 2024-03-25T18:30:52.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937659172 0 0) 0x7fed0c003520 con 0x7fed2414be90 2024-03-25T18:30:52.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 >> v1:172.21.15.42:6790/0 conn(0x7fed240b4d60 legacy=0x7fed2413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:52.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 >> v1:172.21.15.42:6789/0 conn(0x7fed240b1730 legacy=0x7fed240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:52.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:52.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 1 monclient: found mon.noname-c 2024-03-25T18:30:52.920 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240b1030 con 0x7fed2414be90 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fed1c001a10 con 0x7fed2414be90 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed227fc640 10 monclient: dump: 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:52.921 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:52.922 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:52.922 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:52.922 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:52.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed29df6640 5 monclient: authenticate success, global_id 4741 2024-03-25T18:30:52.922 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.919+0000 7fed29df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:52.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed1c001b70 con 0x7fed2414be90 2024-03-25T18:30:52.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:52.952 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed227fc640 1 -- 172.21.15.42:0/4268853962 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fed1c001f80 con 0x7fed2414be90 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed227fc640 10 monclient: dump: 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed29df6640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:52.953 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.951+0000 7fed29df6640 4 set_mon_vals no callback set 2024-03-25T18:30:52.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 10 monclient: shutdown 2024-03-25T18:30:52.955 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 1 -- 172.21.15.42:0/4268853962 >> v1:172.21.15.120:6789/0 conn(0x7fed2414be90 legacy=0x7fed2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 1 -- 172.21.15.42:0/4268853962 shutdown_connections 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 1 -- 172.21.15.42:0/4268853962 >> 172.21.15.42:0/4268853962 conn(0x7fed240617a0 msgr2=0x7fed24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 1 -- 172.21.15.42:0/4268853962 shutdown_connections 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 1 -- 172.21.15.42:0/4268853962 wait complete. 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 10 monclient: build_initial_monmap 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.952+0000 7fed29df6640 10 monclient: monmap: 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:30:52.956 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.953+0000 7fed29df6640 1 librados: starting msgr at 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.953+0000 7fed29df6640 1 librados: starting objecter 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.953+0000 7fed29df6640 1 Processor -- start 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.953+0000 7fed29df6640 1 -- start start 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.953+0000 7fed29df6640 1 librados: setting wanted keys 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.953+0000 7fed29df6640 1 librados: calling monclient init 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.953+0000 7fed29df6640 10 monclient: init 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 10 monclient(hunting): picked mon.noname-a con 0x7fed240b1730 addr v1:172.21.15.42:6789/0 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 10 monclient(hunting): picked mon.noname-b con 0x7fed240b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 10 monclient(hunting): picked mon.noname-c con 0x7fed2414be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed24143900 con 0x7fed240b1730 2024-03-25T18:30:52.957 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed24137de0 con 0x7fed240b4d60 2024-03-25T18:30:52.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed24138050 con 0x7fed2414be90 2024-03-25T18:30:52.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:52.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.954+0000 7fed29df6640 10 monclient(hunting): authenticate will time out at 7968.180790s 2024-03-25T18:30:52.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.956+0000 7fed22ffd640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fed240b4d60 0x7fed24143d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:46510/0 (socket says 172.21.15.42:46510) 2024-03-25T18:30:52.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.956+0000 7fed22ffd640 1 -- 172.21.15.42:0/2873618394 learned_addr learned my addr 172.21.15.42:0/2873618394 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:30:52.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.957+0000 7fed0bfff640 10 client.?.objecter ms_handle_connect 0x7fed240b4d60 2024-03-25T18:30:52.958 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.957+0000 7fed0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:52.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811016420 0 0) 0x7fed24137de0 con 0x7fed240b4d60 2024-03-25T18:30:52.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:52.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:52.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:52.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:30:52.969 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10004eb0 con 0x7fed240b4d60 2024-03-25T18:30:52.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 10 client.?.objecter ms_handle_connect 0x7fed2414be90 2024-03-25T18:30:52.970 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.968+0000 7fed0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:52.976 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.974+0000 7fed0bfff640 10 client.?.objecter ms_handle_connect 0x7fed240b1730 2024-03-25T18:30:52.976 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.975+0000 7fed0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:52.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.979+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606782238 0 0) 0x7fed24138050 con 0x7fed2414be90 2024-03-25T18:30:52.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.979+0000 7fed0bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:52.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.979+0000 7fed0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:52.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.979+0000 7fed0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:52.980 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.979+0000 7fed0bfff640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:30:52.981 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.979+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed24137de0 con 0x7fed2414be90 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.981+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2916305572 0 0) 0x7fed10004eb0 con 0x7fed240b4d60 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.981+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed24138050 con 0x7fed240b4d60 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.981+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fed18001a10 con 0x7fed240b4d60 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.981+0000 7fed0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.981+0000 7fed0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.981+0000 7fed0bfff640 10 monclient(hunting): dump: 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:52.983 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:52.984 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:52.984 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:52.984 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:52.984 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.981+0000 7fed0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1453455167 0 0) 0x7fed24138050 con 0x7fed240b4d60 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 >> v1:172.21.15.42:6790/0 conn(0x7fed2414be90 legacy=0x7fed241444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 >> v1:172.21.15.42:6789/0 conn(0x7fed240b1730 legacy=0x7fed241db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 1 monclient: found mon.b 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed241dbb10 con 0x7fed240b4d60 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 10 monclient: _finish_auth 0 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 5 monclient: authenticate success, global_id 4744 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 10 monclient: _renew_subs 2024-03-25T18:30:52.990 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:52.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed241dbd60 con 0x7fed240b4d60 2024-03-25T18:30:52.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 10 client.4744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:52.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 10 monclient: _renew_subs 2024-03-25T18:30:52.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:52.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed241e38c0 con 0x7fed240b4d60 2024-03-25T18:30:52.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.988+0000 7fed29df6640 1 librados: init done 2024-03-25T18:30:52.991 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.989+0000 7fed29df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:52.992 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.989+0000 7fed29df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:52.992 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.989+0000 7fed29df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:52.992 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:52.989+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed241392d0 con 0x7fed240b4d60 2024-03-25T18:30:53.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.257+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed18001c50 con 0x7fed240b4d60 2024-03-25T18:30:53.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.257+0000 7fed0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:53.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.257+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fed180038c0 con 0x7fed240b4d60 2024-03-25T18:30:53.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.257+0000 7fed0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:53.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.257+0000 7fed0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:53.259 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.257+0000 7fed0bfff640 10 monclient: dump: 2024-03-25T18:30:53.259 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:30:53.260 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.257+0000 7fed21ffb640 4 set_mon_vals no callback set 2024-03-25T18:30:53.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.258+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fed18003ae0 con 0x7fed240b4d60 2024-03-25T18:30:53.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.259+0000 7fed237fe640 10 monclient: get_auth_request con 0x7fed1006dd10 auth_method 0 2024-03-25T18:30:53.261 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.259+0000 7fed0bfff640 10 client.4744.objecter ms_handle_connect 0x7fed1006dd10 2024-03-25T18:30:53.264 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7fed180546d0 con 0x7fed240b4d60 2024-03-25T18:30:53.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 10 client.4744.objecter ms_dispatch 0x7fed24000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:53.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 3 client.4744.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:53.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 3 client.4744.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:53.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 20 client.4744.objecter dump_active .. 0 homeless 2024-03-25T18:30:53.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed180501e0 con 0x7fed240b4d60 2024-03-25T18:30:53.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:53.265 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.263+0000 7fed0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:53.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.506+0000 7fed29df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:53.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.506+0000 7fed29df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:53.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.506+0000 7fed29df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:30:53.508 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.506+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed240627b0 con 0x7fed240b4d60 2024-03-25T18:30:53.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.523+0000 7fed0bfff640 1 -- 172.21.15.42:0/2873618394 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7fed18052110 con 0x7fed240b4d60 2024-03-25T18:30:53.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.523+0000 7fed0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:53.525 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.523+0000 7fed0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:53.525 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:30:53.525 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 10 librados: watch_flush enter 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 10 librados: watch_flush exit 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 20 client.4744.objecter shutdown clearing up homeless session... 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 10 client.4744.objecter successfully canceled tick 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 >> v1:172.21.15.42:6800/3081750888 conn(0x7fed1006dd10 legacy=0x7fed100701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 10 monclient: shutdown 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:53.527 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 >> v1:172.21.15.120:6789/0 conn(0x7fed240b4d60 legacy=0x7fed24143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:53.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 shutdown_connections 2024-03-25T18:30:53.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.525+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 >> 172.21.15.42:0/2873618394 conn(0x7fed240617a0 msgr2=0x7fed2413c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:53.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.526+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 shutdown_connections 2024-03-25T18:30:53.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.526+0000 7fed29df6640 1 -- 172.21.15.42:0/2873618394 wait complete. 2024-03-25T18:30:53.528 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:30:53.526+0000 7fed29df6640 1 librados: shutdown 2024-03-25T18:30:53.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:53.654 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:36.823 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.662 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:36.832 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.685 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:36.854 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.710 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:36.830 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.719 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:36.888 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.723 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:36.843 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.731 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:36.901 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.751 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:36.871 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:53.757 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:36.927 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:00.344 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.463+0000 7fd536967640 10 monclient: get_monmap_and_config 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.463+0000 7fd536967640 10 monclient: build_initial_monmap 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.463+0000 7fd536967640 10 monclient: monmap: 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:31:00.464839+0000 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:31:00.464839+0000 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:31:00.465 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:31:00.466 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.464+0000 7fd536967640 1 Processor -- start 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.464+0000 7fd536967640 1 -- start start 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.464+0000 7fd536967640 10 monclient: init 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 10 monclient(hunting): picked mon.noname-a con 0x7fd5300b4d60 addr v1:172.21.15.42:6789/0 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 10 monclient(hunting): picked mon.noname-c con 0x7fd5300b1730 addr v1:172.21.15.120:6789/0 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 10 monclient(hunting): picked mon.noname-b con 0x7fd53014be90 addr v1:172.21.15.42:6790/0 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300af940 con 0x7fd5300b4d60 2024-03-25T18:31:00.467 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300b0a70 con 0x7fd5300b1730 2024-03-25T18:31:00.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300b0d50 con 0x7fd53014be90 2024-03-25T18:31:00.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:00.468 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.465+0000 7fd536967640 10 monclient(hunting): authenticate will time out at 7975.692092s 2024-03-25T18:31:00.484 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.483+0000 7fd52f7fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd5300b1730 0x7fd5300b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:56422/0 (socket says 172.21.15.42:56422) 2024-03-25T18:31:00.484 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.483+0000 7fd52f7fe640 1 -- 172.21.15.42:0/1175900631 learned_addr learned my addr 172.21.15.42:0/1175900631 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:31:00.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.490+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116635633 0 0) 0x7fd5300b0a70 con 0x7fd5300b1730 2024-03-25T18:31:00.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.490+0000 7fd52effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:00.491 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.490+0000 7fd52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:00.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.490+0000 7fd52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:00.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.490+0000 7fd52effd640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:31:00.492 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.490+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd518003520 con 0x7fd5300b1730 2024-03-25T18:31:00.506 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.505+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444377275 0 0) 0x7fd5300af940 con 0x7fd5300b4d60 2024-03-25T18:31:00.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.505+0000 7fd52effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:00.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.505+0000 7fd52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:00.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.505+0000 7fd52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:00.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.505+0000 7fd52effd640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:31:00.507 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.505+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5300b0a70 con 0x7fd5300b4d60 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039202898 0 0) 0x7fd518003520 con 0x7fd5300b1730 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 >> v1:172.21.15.42:6790/0 conn(0x7fd53014be90 legacy=0x7fd53014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 >> v1:172.21.15.42:6789/0 conn(0x7fd5300b4d60 legacy=0x7fd53013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 1 monclient: found mon.noname-c 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300b1030 con 0x7fd5300b1730 2024-03-25T18:31:00.510 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 10 monclient: _finish_auth 0 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd524001690 con 0x7fd5300b1730 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.508+0000 7fd52effd640 10 monclient: dump: 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:00.511 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.509+0000 7fd536967640 5 monclient: authenticate success, global_id 4747 2024-03-25T18:31:00.512 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.509+0000 7fd536967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:00.517 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5240017f0 con 0x7fd5300b1730 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd52effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd52effd640 1 -- 172.21.15.42:0/1175900631 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd524001c00 con 0x7fd5300b1730 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd52effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd52effd640 10 monclient: dump: 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:00.518 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:00.519 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:00.519 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:00.519 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd536967640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:00.519 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.516+0000 7fd536967640 4 set_mon_vals no callback set 2024-03-25T18:31:00.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.517+0000 7fd536967640 10 monclient: shutdown 2024-03-25T18:31:00.520 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.517+0000 7fd536967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.517+0000 7fd536967640 1 -- 172.21.15.42:0/1175900631 >> v1:172.21.15.120:6789/0 conn(0x7fd5300b1730 legacy=0x7fd5300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.517+0000 7fd536967640 1 -- 172.21.15.42:0/1175900631 shutdown_connections 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.517+0000 7fd536967640 1 -- 172.21.15.42:0/1175900631 >> 172.21.15.42:0/1175900631 conn(0x7fd5300617a0 msgr2=0x7fd530061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.517+0000 7fd536967640 1 -- 172.21.15.42:0/1175900631 shutdown_connections 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.517+0000 7fd536967640 1 -- 172.21.15.42:0/1175900631 wait complete. 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.518+0000 7fd536967640 10 monclient: build_initial_monmap 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.518+0000 7fd536967640 10 monclient: monmap: 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:31:00.521 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.519+0000 7fd536967640 1 librados: starting msgr at 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.519+0000 7fd536967640 1 librados: starting objecter 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.519+0000 7fd536967640 1 Processor -- start 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.519+0000 7fd536967640 1 -- start start 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.519+0000 7fd536967640 1 librados: setting wanted keys 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.519+0000 7fd536967640 1 librados: calling monclient init 2024-03-25T18:31:00.522 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.519+0000 7fd536967640 10 monclient: init 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 10 monclient(hunting): picked mon.noname-c con 0x7fd5300b1730 addr v1:172.21.15.42:6790/0 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 10 monclient(hunting): picked mon.noname-b con 0x7fd5300b4d60 addr v1:172.21.15.120:6789/0 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 10 monclient(hunting): picked mon.noname-a con 0x7fd53014be90 addr v1:172.21.15.42:6789/0 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd530141d50 con 0x7fd53014be90 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5301d8620 con 0x7fd5300b4d60 2024-03-25T18:31:00.523 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5301d79f0 con 0x7fd5300b1730 2024-03-25T18:31:00.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:00.524 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.520+0000 7fd536967640 10 monclient(hunting): authenticate will time out at 7975.747068s 2024-03-25T18:31:00.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.527+0000 7fd52f7fe640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd5300b4d60 0x7fd5301405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:56446/0 (socket says 172.21.15.42:56446) 2024-03-25T18:31:00.529 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.527+0000 7fd52f7fe640 1 -- 172.21.15.42:0/3387920386 learned_addr learned my addr 172.21.15.42:0/3387920386 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:31:00.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.530+0000 7fd52cff9640 10 client.?.objecter ms_handle_connect 0x7fd5300b4d60 2024-03-25T18:31:00.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.530+0000 7fd52cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:00.531 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.531+0000 7fd52cff9640 10 client.?.objecter ms_handle_connect 0x7fd53014be90 2024-03-25T18:31:00.532 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.531+0000 7fd52cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:00.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.533+0000 7fd52cff9640 10 client.?.objecter ms_handle_connect 0x7fd5300b1730 2024-03-25T18:31:00.535 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.533+0000 7fd52cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:00.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.540+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569965192 0 0) 0x7fd530141d50 con 0x7fd53014be90 2024-03-25T18:31:00.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.540+0000 7fd52cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:00.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.540+0000 7fd52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:00.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.540+0000 7fd52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:00.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.540+0000 7fd52cff9640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:31:00.541 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.540+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c004eb0 con 0x7fd53014be90 2024-03-25T18:31:00.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.548+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373868569 0 0) 0x7fd5301d79f0 con 0x7fd5300b1730 2024-03-25T18:31:00.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.548+0000 7fd52cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:00.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.548+0000 7fd52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:00.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.548+0000 7fd52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:00.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.548+0000 7fd52cff9640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:31:00.550 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.548+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd530141d50 con 0x7fd5300b1730 2024-03-25T18:31:00.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.553+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569456633 0 0) 0x7fd5301d8620 con 0x7fd5300b4d60 2024-03-25T18:31:00.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.553+0000 7fd52cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:00.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.553+0000 7fd52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:00.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.553+0000 7fd52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:00.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.553+0000 7fd52cff9640 10 monclient(hunting): my global_id is 4750 2024-03-25T18:31:00.554 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.553+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5301d79f0 con 0x7fd5300b4d60 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.560+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2951201905 0 0) 0x7fd51c004eb0 con 0x7fd53014be90 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.561+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5301d8620 con 0x7fd53014be90 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.561+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd528001b90 con 0x7fd53014be90 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.561+0000 7fd52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.561+0000 7fd52cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.561+0000 7fd52cff9640 10 monclient(hunting): dump: 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.562 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:00.563 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:00.563 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:00.563 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:00.563 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:00.563 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.563 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.561+0000 7fd52cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.563+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2770278147 0 0) 0x7fd5301d79f0 con 0x7fd5300b4d60 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.563+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd51c004eb0 con 0x7fd5300b4d60 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.563+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd524001b00 con 0x7fd5300b4d60 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.563+0000 7fd52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.563+0000 7fd52cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.563+0000 7fd52cff9640 10 monclient(hunting): dump: 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:00.564 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:00.565 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.569+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.2 v1:172.21.15.42:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1602431044 0 0) 0x7fd530141d50 con 0x7fd5300b1730 2024-03-25T18:31:00.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.569+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5301d79f0 con 0x7fd5300b1730 2024-03-25T18:31:00.570 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.569+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.2 v1:172.21.15.42:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd520001930 con 0x7fd5300b1730 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.569+0000 7fd52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.569+0000 7fd52cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.569+0000 7fd52cff9640 10 monclient(hunting): dump: 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:00.571 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.583 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3406174448 0 0) 0x7fd5301d8620 con 0x7fd53014be90 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 >> v1:172.21.15.42:6790/0 conn(0x7fd5300b1730 legacy=0x7fd5301d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 >> v1:172.21.15.120:6789/0 conn(0x7fd5300b4d60 legacy=0x7fd5301405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 1 monclient: found mon.a 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530140d00 con 0x7fd53014be90 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 10 monclient: _finish_auth 0 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd52cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 5 monclient: authenticate success, global_id 4734 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 10 monclient: _renew_subs 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:31:00.584 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53013b630 con 0x7fd53014be90 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 10 monclient: _renew_subs 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53013b8e0 con 0x7fd53014be90 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.582+0000 7fd536967640 1 librados: init done 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.583+0000 7fd536967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.583+0000 7fd536967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.583+0000 7fd536967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:31:00.585 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.583+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5301d7830 con 0x7fd53014be90 2024-03-25T18:31:00.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.861+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd528002080 con 0x7fd53014be90 2024-03-25T18:31:00.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.861+0000 7fd52cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:00.862 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.861+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7fd528003b10 con 0x7fd53014be90 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.861+0000 7fd52cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.861+0000 7fd52cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.861+0000 7fd52cff9640 10 monclient: dump: 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:00.863 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:00.864 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:00.864 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.861+0000 7fd52e7fc640 4 set_mon_vals no callback set 2024-03-25T18:31:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.862+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7fd528003d30 con 0x7fd53014be90 2024-03-25T18:31:00.864 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.862+0000 7fd52ffff640 10 monclient: get_auth_request con 0x7fd51c06e0b0 auth_method 0 2024-03-25T18:31:00.865 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.862+0000 7fd52cff9640 10 client.4734.objecter ms_handle_connect 0x7fd51c06e0b0 2024-03-25T18:31:00.913 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7fd5280537f0 con 0x7fd53014be90 2024-03-25T18:31:00.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 10 client.4734.objecter ms_dispatch 0x7fd530000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:31:00.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 3 client.4734.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:31:00.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 3 client.4734.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:31:00.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 20 client.4734.objecter dump_active .. 0 homeless 2024-03-25T18:31:00.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd52804f130 con 0x7fd53014be90 2024-03-25T18:31:00.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:00.914 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:00.912+0000 7fd52cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:01.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.176+0000 7fd536967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:01.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.176+0000 7fd536967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:01.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.176+0000 7fd536967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.42:6789/0 2024-03-25T18:31:01.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.176+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 --> v1:172.21.15.42:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5301d7b80 con 0x7fd53014be90 2024-03-25T18:31:01.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.229+0000 7fd52cff9640 1 -- 172.21.15.42:0/3387920386 <== mon.0 v1:172.21.15.42:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7fd528051230 con 0x7fd53014be90 2024-03-25T18:31:01.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.229+0000 7fd52cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:01.231 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.229+0000 7fd52cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:01.231 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:31:01.231 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 10 librados: watch_flush enter 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 10 librados: watch_flush exit 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 20 client.4734.objecter shutdown clearing up homeless session... 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 10 client.4734.objecter successfully canceled tick 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 >> v1:172.21.15.42:6800/3081750888 conn(0x7fd51c06e0b0 legacy=0x7fd51c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 10 monclient: shutdown 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:01.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 >> v1:172.21.15.42:6789/0 conn(0x7fd53014be90 legacy=0x7fd530142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:01.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 shutdown_connections 2024-03-25T18:31:01.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 >> 172.21.15.42:0/3387920386 conn(0x7fd5300617a0 msgr2=0x7fd530138590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:01.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.232+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 shutdown_connections 2024-03-25T18:31:01.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.233+0000 7fd536967640 1 -- 172.21.15.42:0/3387920386 wait complete. 2024-03-25T18:31:01.235 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:01.233+0000 7fd536967640 1 librados: shutdown 2024-03-25T18:31:01.246 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:08.053 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:08.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.167+0000 7f6ca3d2b640 10 monclient: get_monmap_and_config 2024-03-25T18:31:08.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.167+0000 7f6ca3d2b640 10 monclient: build_initial_monmap 2024-03-25T18:31:08.168 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.167+0000 7f6ca3d2b640 10 monclient: monmap: 2024-03-25T18:31:08.168 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:31:08.168 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:08.168 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:31:08.168347+0000 2024-03-25T18:31:08.169 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:31:08.168347+0000 2024-03-25T18:31:08.169 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:08.169 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:08.169 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:31:08.169 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-c 2024-03-25T18:31:08.169 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-b 2024-03-25T18:31:08.169 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.168+0000 7f6ca3d2b640 1 Processor -- start 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.168+0000 7f6ca3d2b640 1 -- start start 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.168+0000 7f6ca3d2b640 10 monclient: init 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.168+0000 7f6ca3d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9c0a98f0 addr v1:172.21.15.42:6789/0 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9c13f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9c1430c0 addr v1:172.21.15.120:6789/0 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c142520 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c0b28a0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c0b2a30 con 0x7f6c9c13f710 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:08.171 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.169+0000 7f6ca3d2b640 10 monclient(hunting): authenticate will time out at 7983.395681s 2024-03-25T18:31:08.176 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.175+0000 7f6ca22c8640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f6c9c1430c0 0x7f6c9c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.42:34394/0 (socket says 172.21.15.42:34394) 2024-03-25T18:31:08.177 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.175+0000 7f6ca22c8640 1 -- 172.21.15.42:0/2109118265 learned_addr learned my addr 172.21.15.42:0/2109118265 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:31:08.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.183+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654457702 0 0) 0x7f6c9c142520 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.184+0000 7f6ca0ac5640 10 monclient(hunting): _negotiate 2024-03-25T18:31:08.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.184+0000 7f6ca0ac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:08.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.184+0000 7f6ca0ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:08.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.184+0000 7f6ca0ac5640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:31:08.185 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.184+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c88003520 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.186+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180575348 0 0) 0x7f6c9c0b28a0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.186+0000 7f6ca0ac5640 10 monclient(hunting): _negotiate 2024-03-25T18:31:08.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.186+0000 7f6ca0ac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:08.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.186+0000 7f6ca0ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:08.187 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.186+0000 7f6ca0ac5640 10 monclient(hunting): my global_id is 4753 2024-03-25T18:31:08.188 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.186+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c142520 con 0x7f6c9c1430c0 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1114002894 0 0) 0x7f6c9c142520 con 0x7f6c9c1430c0 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 >> v1:172.21.15.42:6790/0 conn(0x7f6c9c13f710 legacy=0x7f6c9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 >> v1:172.21.15.42:6789/0 conn(0x7f6c9c0a98f0 legacy=0x7f6c9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 10 monclient: _finish_hunting 0 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 1 monclient: found mon.noname-c 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.120:6789/0 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c0b44c0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 10 monclient: _finish_auth 0 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 10 monclient: _check_auth_tickets 2024-03-25T18:31:08.199 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6c940019f0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca0ac5640 10 monclient: dump: 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:08.200 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:08.201 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:08.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca3d2b640 5 monclient: authenticate success, global_id 4753 2024-03-25T18:31:08.201 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.197+0000 7f6ca3d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:08.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 <== mon.1 v1:172.21.15.120:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c94001b50 con 0x7f6c9c1430c0 2024-03-25T18:31:08.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca0ac5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:08.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca0ac5640 1 -- 172.21.15.42:0/2109118265 <== mon.1 v1:172.21.15.120:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6c94001f60 con 0x7f6c9c1430c0 2024-03-25T18:31:08.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca0ac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:08.205 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca0ac5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca0ac5640 10 monclient: dump: 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:08.206 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:08.207 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:08.207 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca3d2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:08.207 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.204+0000 7f6ca3d2b640 4 set_mon_vals no callback set 2024-03-25T18:31:08.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.205+0000 7f6ca3d2b640 10 monclient: shutdown 2024-03-25T18:31:08.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.205+0000 7f6ca3d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:08.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.205+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/2109118265 >> v1:172.21.15.120:6789/0 conn(0x7f6c9c1430c0 legacy=0x7f6c9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.209 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.205+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/2109118265 shutdown_connections 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.205+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/2109118265 >> 172.21.15.42:0/2109118265 conn(0x7f6c9c0b7ea0 msgr2=0x7f6c9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.205+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/2109118265 shutdown_connections 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.205+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/2109118265 wait complete. 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 10 monclient: build_initial_monmap 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 10 monclient: monmap: 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-03-25T18:31:08.210 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.noname-b 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.noname-c 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 1 librados: starting msgr at 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 1 librados: starting objecter 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 1 Processor -- start 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 1 -- start start 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 1 librados: setting wanted keys 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 1 librados: calling monclient init 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.206+0000 7f6ca3d2b640 10 monclient: init 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:08.211 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9c0a98f0 addr v1:172.21.15.120:6789/0 2024-03-25T18:31:08.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9c13f710 addr v1:172.21.15.42:6790/0 2024-03-25T18:31:08.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9c1430c0 addr v1:172.21.15.42:6789/0 2024-03-25T18:31:08.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c0b7be0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c13cb80 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 1 -- --> v1:172.21.15.42:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c9c1dc070 con 0x7f6c9c13f710 2024-03-25T18:31:08.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:08.212 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.207+0000 7f6ca3d2b640 10 monclient(hunting): authenticate will time out at 7983.434307s 2024-03-25T18:31:08.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.219+0000 7f6ca22c8640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6c9c1430c0 0x7f6c9c1db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53350/0 (socket says 172.21.15.42:53350) 2024-03-25T18:31:08.220 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.219+0000 7f6ca22c8640 1 -- 172.21.15.42:0/1897338785 learned_addr learned my addr 172.21.15.42:0/1897338785 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-03-25T18:31:08.223 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.222+0000 7f6c927fc640 10 client.?.objecter ms_handle_connect 0x7f6c9c1430c0 2024-03-25T18:31:08.223 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.222+0000 7f6c927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:08.226 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.224+0000 7f6c927fc640 10 client.?.objecter ms_handle_connect 0x7f6c9c0a98f0 2024-03-25T18:31:08.226 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.225+0000 7f6c927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:08.233 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.232+0000 7f6c927fc640 10 client.?.objecter ms_handle_connect 0x7f6c9c13f710 2024-03-25T18:31:08.233 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.232+0000 7f6c927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:08.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.233+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240649396 0 0) 0x7f6c9c0b7be0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.233+0000 7f6c927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:08.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.233+0000 7f6c927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:08.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.233+0000 7f6c927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:08.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.233+0000 7f6c927fc640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:31:08.234 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.233+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c80004eb0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.238 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.237+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100317109 0 0) 0x7f6c9c13cb80 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.238 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.237+0000 7f6c927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:08.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.237+0000 7f6c927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:08.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.237+0000 7f6c927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:08.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.237+0000 7f6c927fc640 10 monclient(hunting): my global_id is 4756 2024-03-25T18:31:08.239 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.237+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c0b7be0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.247+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 668324757 0 0) 0x7f6c80004eb0 con 0x7f6c9c1430c0 2024-03-25T18:31:08.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.247+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c9c13cb80 con 0x7f6c9c1430c0 2024-03-25T18:31:08.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.247+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6c94001f20 con 0x7f6c9c1430c0 2024-03-25T18:31:08.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.247+0000 7f6c927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:08.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.247+0000 7f6c927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:08.248 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.247+0000 7f6c927fc640 10 monclient(hunting): dump: 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:08.249 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:08.250 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:08.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.247+0000 7f6c927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:08.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.248+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754204509 0 0) 0x7f6c9c0b7be0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.248+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.120:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c80004eb0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.250 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.248+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6c98001950 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.248+0000 7f6c927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.248+0000 7f6c927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.248+0000 7f6c927fc640 10 monclient(hunting): dump: 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:08.251 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:08.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.255+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.2 v1:172.21.15.42:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533551956 0 0) 0x7f6c9c1dc070 con 0x7f6c9c13f710 2024-03-25T18:31:08.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.255+0000 7f6c927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:08.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.255+0000 7f6c927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:08.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.255+0000 7f6c927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:08.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.255+0000 7f6c927fc640 10 monclient(hunting): my global_id is 4748 2024-03-25T18:31:08.257 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.255+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.42:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c0b7be0 con 0x7f6c9c13f710 2024-03-25T18:31:08.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 949892428 0 0) 0x7f6c80004eb0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 >> v1:172.21.15.42:6790/0 conn(0x7f6c9c13f710 legacy=0x7f6c9c1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 >> v1:172.21.15.42:6789/0 conn(0x7f6c9c1430c0 legacy=0x7f6c9c1db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:08.271 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 1 monclient: found mon.b 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c13b740 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6c927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6ca3d2b640 5 monclient: authenticate success, global_id 4756 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6ca3d2b640 10 monclient: _renew_subs 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6ca3d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.269+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.120:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c130410 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.270+0000 7f6ca3d2b640 10 client.4756.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.270+0000 7f6ca3d2b640 10 monclient: _renew_subs 2024-03-25T18:31:08.272 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.270+0000 7f6ca3d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:31:08.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.270+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.120:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c14e6e0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.270+0000 7f6ca3d2b640 1 librados: init done 2024-03-25T18:31:08.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.271+0000 7f6ca3d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:08.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.271+0000 7f6ca3d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:08.273 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.271+0000 7f6ca3d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:31:08.274 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.271+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c0b3ef0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.551+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c98001d60 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.551+0000 7f6c927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:08.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.551+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3767129501 0 0) 0x7f6c980037b0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.551+0000 7f6c927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:08.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.551+0000 7f6c927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:08.553 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.551+0000 7f6c927fc640 10 monclient: dump: 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:fsid 402856df-369d-4d95-b0d1-b534080fb192 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:created 2024-03-25T18:18:45.855670+0000 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 3 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.a 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.120:6789/0 mon.b 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr:2: v1:172.21.15.42:6790/0 mon.c 2024-03-25T18:31:08.554 INFO:teuthology.misc.health.smithi042.stderr: 2024-03-25T18:31:08.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.551+0000 7f6c93fff640 4 set_mon_vals no callback set 2024-03-25T18:31:08.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.552+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3516074814 0 0) 0x7f6c980039d0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.552+0000 7f6ca12c6640 10 monclient: get_auth_request con 0x7f6c8006df00 auth_method 0 2024-03-25T18:31:08.555 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.552+0000 7f6c927fc640 10 client.4756.objecter ms_handle_connect 0x7f6c8006df00 2024-03-25T18:31:08.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2844113294 0 0) 0x7f6c980534d0 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 10 client.4756.objecter ms_dispatch 0x7f6c9c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:31:08.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 3 client.4756.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:31:08.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 3 client.4756.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:31:08.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 20 client.4756.objecter dump_active .. 0 homeless 2024-03-25T18:31:08.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c9804ee10 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.559 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:08.560 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.557+0000 7f6c927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:08.798 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.797+0000 7f6ca3d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:08.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.797+0000 7f6ca3d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:08.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.797+0000 7f6ca3d2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.120:6789/0 2024-03-25T18:31:08.799 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.797+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 --> v1:172.21.15.120:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c9c14f330 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.825+0000 7f6c927fc640 1 -- 172.21.15.42:0/1897338785 <== mon.1 v1:172.21.15.120:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2555081543) 0x7f6c9c14f330 con 0x7f6c9c0a98f0 2024-03-25T18:31:08.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.825+0000 7f6c927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:08.827 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.825+0000 7f6c927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:08.827 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-25T18:31:08.827 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":11,"fsid":"402856df-369d-4d95-b0d1-b534080fb192","created":"2024-03-25T18:19:39.417083+0000","modified":"2024-03-25T18:29:43.338780+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:43.338780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"300de658-80a1-43d5-91bb-a6be671335e8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"76fae7c0-88cd-4858-898c-ee21e1b87701","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a44eba7f-b7c9-4441-8561-ef78dafca07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"408e355a-5bb9-4fde-8fb3-c5256132c2ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e53f94b-2a33-459d-b200-fd4fe94ea86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d98d0642-875c-461c-80aa-aa17bdd13517","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6dbbddf8-9fe5-48c5-b3fc-3bf444381159","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"01b63363-95ba-4457-9474-00febf9e8452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:08.829 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 10 librados: watch_flush enter 2024-03-25T18:31:08.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 10 librados: watch_flush exit 2024-03-25T18:31:08.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 20 client.4756.objecter shutdown clearing up homeless session... 2024-03-25T18:31:08.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 10 client.4756.objecter successfully canceled tick 2024-03-25T18:31:08.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 >> v1:172.21.15.42:6800/3081750888 conn(0x7f6c8006df00 legacy=0x7f6c800703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 10 monclient: shutdown 2024-03-25T18:31:08.830 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:08.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.827+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 >> v1:172.21.15.120:6789/0 conn(0x7f6c9c0a98f0 legacy=0x7f6c9c139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:08.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.828+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 shutdown_connections 2024-03-25T18:31:08.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.828+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 >> 172.21.15.42:0/1897338785 conn(0x7f6c9c0b7ea0 msgr2=0x7f6c9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:08.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.828+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 shutdown_connections 2024-03-25T18:31:08.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.828+0000 7f6ca3d2b640 1 -- 172.21.15.42:0/1897338785 wait complete. 2024-03-25T18:31:08.831 INFO:teuthology.misc.health.smithi042.stderr:2024-03-25T18:31:08.828+0000 7f6ca3d2b640 1 librados: shutdown 2024-03-25T18:31:08.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:08.843 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:31:08.845 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:31:08.845 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:31:08.845 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:10.912 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:11:25.985 35410== 2024-03-25T18:31:10.912 INFO:tasks.ceph.osd.0.smithi042.stderr:==00:00:11:25.985 35410== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:10.951 INFO:tasks.ceph.osd.0.smithi042.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:13.688 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:56.857 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.740 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:56.909 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.784 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:11:56.954 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.786 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:56.906 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.798 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:56.918 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.822 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:11:56.942 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.825 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:56.995 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.831 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:57.000 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:13.870 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:11:57.040 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:14.948 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:31:14.949 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:31:14.949 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:31:14.949 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:16.974 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:11:32.029 35412== 2024-03-25T18:31:16.974 INFO:tasks.ceph.osd.1.smithi042.stderr:==00:00:11:32.029 35412== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:17.014 INFO:tasks.ceph.osd.1.smithi042.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:21.051 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:31:21.051 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:31:21.051 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:31:21.052 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:23.021 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:11:38.093 35411== 2024-03-25T18:31:23.021 INFO:tasks.ceph.osd.2.smithi042.stderr:==00:00:11:38.093 35411== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:23.060 INFO:tasks.ceph.osd.2.smithi042.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:27.153 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:31:27.154 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:31:27.154 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:31:27.154 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:28.938 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:11:43.990 35413== 2024-03-25T18:31:28.938 INFO:tasks.ceph.osd.3.smithi042.stderr:==00:00:11:43.990 35413== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:28.980 INFO:tasks.ceph.osd.3.smithi042.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:33.257 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:31:33.257 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:31:33.257 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:31:33.257 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:33.774 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:12:16.941 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.777 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:12:16.947 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.802 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:12:16.972 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.827 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:12:16.948 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.829 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:12:16.998 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.830 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:12:16.950 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.833 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:12:17.003 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.848 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:12:16.969 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:33.849 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:12:17.019 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:35.326 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:11:50.395 35419== 2024-03-25T18:31:35.326 INFO:tasks.ceph.osd.4.smithi120.stderr:==00:00:11:50.395 35419== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:35.364 INFO:tasks.ceph.osd.4.smithi120.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:39.360 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:31:39.360 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:31:39.361 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:31:39.361 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:41.320 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:11:56.375 35421== 2024-03-25T18:31:41.320 INFO:tasks.ceph.osd.5.smithi120.stderr:==00:00:11:56.375 35421== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:41.359 INFO:tasks.ceph.osd.5.smithi120.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:45.463 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:31:45.463 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:31:45.463 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:31:45.463 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:47.175 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:12:02.241 35420== 2024-03-25T18:31:47.175 INFO:tasks.ceph.osd.6.smithi120.stderr:==00:00:12:02.241 35420== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:47.217 INFO:tasks.ceph.osd.6.smithi120.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:51.565 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:31:51.565 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:31:51.566 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:31:51.566 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:53.128 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:12:08.173 35422== 2024-03-25T18:31:53.128 INFO:tasks.ceph.osd.7.smithi120.stderr:==00:00:12:08.173 35422== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:53.158 INFO:tasks.ceph.osd.7.smithi120.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:53.835 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:12:37.004 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.839 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:12:37.008 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.850 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:12:37.020 35095== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.881 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:12:37.001 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.882 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:12:37.002 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.886 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:12:37.056 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.889 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:12:37.059 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.894 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:12:37.014 34875== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:53.902 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:12:37.071 35099== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:57.667 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:31:57.668 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:31:57.668 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:31:57.668 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:31:57.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:57.669 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:31:57.668+0000 7f8cc2641640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35104) UID: 0 2024-03-25T18:31:57.669 INFO:tasks.ceph.mgr.y.smithi042.stderr:2024-03-25T18:31:57.668+0000 7f8cc2641640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:31:57.695 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:31:57.695 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:31:57.695 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:57.721 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:31:57.722 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:31:57.722 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:31:57.722 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:57.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-03-25T18:31:57.738+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35064) UID: 0 2024-03-25T18:31:57.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-03-25T18:31:57.740+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:35:02.245 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:15:45.414 35095== 2024-03-25T18:35:02.246 INFO:tasks.ceph.mon.a.smithi042.stderr:==00:00:15:45.414 35095== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:35:02.255 INFO:tasks.ceph.mon.a.smithi042.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:35:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:35:06.864 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:35:06.865 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:35:06.865 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:35:06.865 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:35:06.882 INFO:tasks.ceph.mon.c.smithi042.stderr:2024-03-25T18:35:06.869+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35066) UID: 0 2024-03-25T18:35:06.882 INFO:tasks.ceph.mon.c.smithi042.stderr:2024-03-25T18:35:06.870+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:35:07.482 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:15:50.652 35099== 2024-03-25T18:35:07.482 INFO:tasks.ceph.mon.c.smithi042.stderr:==00:00:15:50.652 35099== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:35:07.493 INFO:tasks.ceph.mon.c.smithi042.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:35:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:35:12.967 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi042 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:35:12.967 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:35:12.968 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:35:12.968 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:35:12.984 INFO:tasks.ceph.mon.b.smithi120.stderr:2024-03-25T18:35:12.971+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34871) UID: 0 2024-03-25T18:35:12.984 INFO:tasks.ceph.mon.b.smithi120.stderr:2024-03-25T18:35:12.973+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:35:13.412 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:15:56.532 34875== 2024-03-25T18:35:13.412 INFO:tasks.ceph.mon.b.smithi120.stderr:==00:00:15:56.532 34875== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:35:13.422 INFO:tasks.ceph.mon.b.smithi120.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:35:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:35:19.070 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:35:19.070 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:35:19.070 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:35:19.071 DEBUG:teuthology.orchestra.run.smithi042:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:35:19.118 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:35:19.118 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:35:19.176 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:35:19.176 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:35:19.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:35:19.260 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:35:19.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:35:19.309 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:35:19.359 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:35:19.359 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:35:19.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:35:19.471 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:35:19.526 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:35:19.526 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:35:19.585 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:35:19.585 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:35:19.642 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:35:19.644 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/data/mon.a.tgz 2024-03-25T18:35:19.644 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-03-25T18:35:19.660 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.7d3wYty1Wq 2024-03-25T18:35:19.660 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7d3wYty1Wq 2024-03-25T18:35:19.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.7d3wYty1Wq 2024-03-25T18:35:19.855 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.7d3wYty1Wq is 133KB 2024-03-25T18:35:19.878 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.7d3wYty1Wq 2024-03-25T18:35:19.892 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/data/mon.c.tgz 2024-03-25T18:35:19.892 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-03-25T18:35:19.946 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.70CZ4Xos5I 2024-03-25T18:35:19.946 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.70CZ4Xos5I 2024-03-25T18:35:20.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.70CZ4Xos5I 2024-03-25T18:35:20.122 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.70CZ4Xos5I is 163KB 2024-03-25T18:35:20.146 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.70CZ4Xos5I 2024-03-25T18:35:20.160 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/data/mon.b.tgz 2024-03-25T18:35:20.160 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-25T18:35:20.178 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.6oLRqbzkcO 2024-03-25T18:35:20.178 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.6oLRqbzkcO 2024-03-25T18:35:20.291 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.6oLRqbzkcO 2024-03-25T18:35:20.378 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.6oLRqbzkcO is 155KB 2024-03-25T18:35:20.401 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.6oLRqbzkcO 2024-03-25T18:35:20.415 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:35:20.416 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:35:20.418 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:35:20.745 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:35:20.929 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:35:20.931 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/remote/smithi042/crash 2024-03-25T18:35:20.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:35:20.959 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/remote/smithi120/crash 2024-03-25T18:35:20.960 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:35:20.986 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:35:20.986 DEBUG:teuthology.orchestra.run.smithi042:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:35:21.003 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:35:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:35:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39425.log 2024-03-25T18:35:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35106.log 2024-03-25T18:35:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-03-25T18:35:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2024-03-25T18:35:21.073 INFO:teuthology.orchestra.run.smithi042.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39425.log.gz 2024-03-25T18:35:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41494.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35106.log.gz 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39815.log 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose 86.2% -- -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz /var/log/ceph/ceph-client.admin.39893.log 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40205.log: /var/log/ceph/ceph-client.admin.39815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42160.log 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41182.log 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39815.log.gz 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42160.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39893.log.gz 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41651.log 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40751.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42160.log.gz 2024-03-25T18:35:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41182.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41651.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41651.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 77.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41104.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-03-25T18:35:21.075 INFO:teuthology.orchestra.run.smithi042.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40283.log 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39308.log 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41104.log.gz 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42002.log 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40283.log.gz 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39659.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39308.log.gz 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39386.log 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41807.log 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42002.log.gz 2024-03-25T18:35:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41963.log 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39269.log 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39386.log.gz 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 86.1% 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41807.log.gz 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41612.log 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41963.log.gz 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40478.log 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39269.log.gz 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41612.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gzgzip 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39152.log 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40478.log: gzip -5 --verbose 86.3% -- -- replaced with /var/log/ceph/ceph-client.admin.41612.log.gz /var/log/ceph/ceph-client.admin.41690.log 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-03-25T18:35:21.077 INFO:teuthology.orchestra.run.smithi042.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40478.log.gz 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39152.log.gz 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41455.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41690.log.gz 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42082.log 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39932.log 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.audit.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42082.log.gz 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39113.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39932.log.gz 2024-03-25T18:35:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39464.log 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40829.log: /var/log/ceph/ceph-client.admin.40946.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40439.log 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40010.log 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39464.log.gz 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40010.log: gzip 87.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz --verbose 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr: -- /var/log/ceph/ceph-client.admin.42238.log 2024-03-25T18:35:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40010.log.gz 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42238.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36543.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42238.log.gz 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.40985.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36543.log.gz 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39698.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40985.log.gz 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40361.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37515.log 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41221.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-03-25T18:35:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39347.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37515.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40361.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37515.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35514.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39347.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35514.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41299.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:35:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40517.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39191.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41846.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39542.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41260.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41846.log.gz 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40088.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39542.log.gz 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39581.log 2024-03-25T18:35:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41924.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39581.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41729.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41260.log.gz 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41924.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39581.log.gz 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41924.log.gz 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40673.log: 87.1%/var/log/ceph/ceph-client.admin.40556.log: -- replaced with /var/log/ceph/ceph-client.admin.41729.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35669.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42199.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35669.log: /var/log/ceph/ceph-client.admin.42199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2024-03-25T18:35:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42199.log.gz 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41572.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35669.log.gz 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39230.log 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40634.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39230.log: gzip -5 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42121.log 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.0.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39230.log.gz 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41533.log 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35409.log 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41533.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42121.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:35:21.084 INFO:teuthology.orchestra.run.smithi042.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41533.log.gz 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40244.log 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35409.log: /var/log/ceph/ceph-client.admin.41416.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35409.log.gz 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40244.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40166.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz/var/log/ceph/ceph.log: 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41377.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40595.log 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29255.log 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40595.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40595.log.gz 2024-03-25T18:35:21.085 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39503.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42316.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39503.log: /var/log/ceph/ceph.tmp-client.admin.29255.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29255.log.gz 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41338.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39503.log.gz 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42316.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42316.log.gz 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41024.log: /var/log/ceph/ceph-client.admin.41338.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41338.log.gz 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39620.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39737.log: /var/log/ceph/ceph-client.admin.39620.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39033.log 2024-03-25T18:35:21.086 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39620.log.gz 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40907.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42041.log 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42041.log: /var/log/ceph/ceph-client.admin.39033.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39033.log.gz 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42041.log.gz 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42277.log: /var/log/ceph/ceph-client.admin.41768.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2024-03-25T18:35:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39971.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-03-25T18:35:21.088 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:35:21.088 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35189.log 2024-03-25T18:35:21.088 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35229.log 2024-03-25T18:35:21.088 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35320.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35229.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35189.log.gz 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35320.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35229.log.gz 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35320.log.gz 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35109.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:35:21.089 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35109.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35269.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35109.log.gz 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35269.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.35269.log.gz 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35069.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35069.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35069.log.gz 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35029.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35149.log 2024-03-25T18:35:21.090 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-mon.b.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35029.log.gz 2024-03-25T18:35:21.091 INFO:teuthology.orchestra.run.smithi120.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:35:21.091 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.35149.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35149.log.gz 2024-03-25T18:35:21.091 INFO:teuthology.orchestra.run.smithi120.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:35:21.128 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-mon.c.log: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:35:21.883 INFO:teuthology.orchestra.run.smithi042.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:35:21.941 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:35:21.970 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:35:21.999 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:35:22.187 INFO:teuthology.orchestra.run.smithi042.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:35:22.216 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:35:22.258 INFO:teuthology.orchestra.run.smithi042.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:35:22.281 INFO:teuthology.orchestra.run.smithi042.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:35:26.068 INFO:teuthology.orchestra.run.smithi042.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:35:26.162 INFO:teuthology.orchestra.run.smithi042.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:35:26.164 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-25T18:35:26.164 INFO:teuthology.orchestra.run.smithi042.stderr:real 0m5.149s 2024-03-25T18:35:26.164 INFO:teuthology.orchestra.run.smithi042.stderr:user 0m14.219s 2024-03-25T18:35:26.164 INFO:teuthology.orchestra.run.smithi042.stderr:sys 0m0.521s 2024-03-25T18:35:26.366 INFO:teuthology.orchestra.run.smithi120.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:35:26.368 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-25T18:35:26.368 INFO:teuthology.orchestra.run.smithi120.stderr:real 0m5.320s 2024-03-25T18:35:26.368 INFO:teuthology.orchestra.run.smithi120.stderr:user 0m8.898s 2024-03-25T18:35:26.368 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m0.316s 2024-03-25T18:35:26.368 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:35:26.368 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/remote/smithi042/log 2024-03-25T18:35:26.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:35:28.632 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/remote/smithi120/log 2024-03-25T18:35:28.633 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:35:30.145 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:35:30.345 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7f756f4e362447a889ef0a707da1d0de Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:35:30.349 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:35:30.358 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:35:30.380 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:35:30.381 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:35:30.394 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:35:30.470 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:35:30.481 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:35:30.481 DEBUG:teuthology.orchestra.run.smithi042:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:35:30.484 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:35:30.497 INFO:teuthology.orchestra.run.smithi042.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:35:30.498 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:35:30.503 INFO:teuthology.orchestra.run.smithi042.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:35:30.503 INFO:teuthology.orchestra.run.smithi042.stdout:=============================================================================== 2024-03-25T18:35:30.503 INFO:teuthology.orchestra.run.smithi042.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +636us[ +636us] +/- 73ms 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi042.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -1056us[-1064us] +/- 58ms 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi042.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +838us[ +838us] +/- 63ms 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +584us[ +655us] +/- 73ms 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -1104us[-1033us] +/- 58ms 2024-03-25T18:35:30.504 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +791us[ +862us] +/- 63ms 2024-03-25T18:35:30.505 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:35:30.505 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:35:30.515 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:35:30.516 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:35:30.526 DEBUG:teuthology.orchestra.run.smithi042:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:35:30.615 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:35:30.681 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:35:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:35:30.712 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:35:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:35:30.741 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:35:30.750 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:35:30.760 INFO:teuthology.task.internal:Duration was 7890.412397 seconds 2024-03-25T18:35:30.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:35:30.770 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:35:30.770 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:35:30.773 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:35:30.815 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:35:30.825 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:35:31.092 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:35:31.092 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com 2024-03-25T18:35:31.092 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:35:31.112 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-03-25T18:35:31.112 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:35:31.159 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:35:31.159 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:35:31.162 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:35:31.229 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:35:31.229 DEBUG:teuthology.orchestra.run.smithi042:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:35:31.231 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:35:31.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:35:31.384 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:35:31.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:35:31.410 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:35:31.437 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:35:31.447 DEBUG:teuthology.orchestra.run.smithi042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:35:31.453 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:35:31.476 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = core 2024-03-25T18:35:31.502 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-03-25T18:35:31.546 DEBUG:teuthology.orchestra.run.smithi042:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:35:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:35:31.559 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:35:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:35:31.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:35:31.610 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:35:31.610 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/remote/smithi042 2024-03-25T18:35:31.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:35:31.645 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272/remote/smithi120 2024-03-25T18:35:31.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:35:31.680 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:35:31.680 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:35:31.683 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:35:31.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:35:31.740 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:35:31.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:35:31.750 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:35:31.750 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:35:31.752 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:35:31.764 INFO:teuthology.orchestra.run.smithi042.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:35 /home/ubuntu/cephtest 2024-03-25T18:35:31.784 INFO:teuthology.orchestra.run.smithi120.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:35 /home/ubuntu/cephtest 2024-03-25T18:35:31.785 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:35:31.795 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:35:31.817 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:35:31.842 DEBUG:teuthology.nuke:shortname: smithi042 2024-03-25T18:35:31.842 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:35:31.864 DEBUG:teuthology.nuke:shortname: smithi120 2024-03-25T18:35:31.864 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:35:31.868 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:15:33.770599', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8BW/raJ7hs8HkgBcrqosAu5QNhsG8VKh8OtuWAIjVOj/xngIyLwFiFziQPbI2DGtRp/gvAIt4u1l82IIosaco='} 2024-03-25T18:35:31.885 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:15:33.769316', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9c2Q9gPXMZsTiZJDGLRFyhiZQQwT0ovlYf6cy2hhSfX+o3b6NEreqMXCwN6ywjkx7gSe/R9PjyQUF8U/yPY4o='} 2024-03-25T18:35:31.910 INFO:teuthology.orchestra.console.smithi042:Power off 2024-03-25T18:35:31.910 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:35:31.934 DEBUG:teuthology.orchestra.console.smithi042:power off output: Chassis Power Control: Down/Off 2024-03-25T18:35:31.934 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:35:31.949 INFO:teuthology.orchestra.console.smithi120:Power off 2024-03-25T18:35:31.949 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:35:31.959 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is on 2024-03-25T18:35:31.972 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off 2024-03-25T18:35:31.972 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:35:31.994 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-25T18:35:35.960 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:35:35.995 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:35:36.071 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is on 2024-03-25T18:35:36.103 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-25T18:35:40.074 DEBUG:teuthology.orchestra.console.smithi042:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:35:40.104 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:35:40.188 DEBUG:teuthology.orchestra.console.smithi042:check power output: Chassis Power is off 2024-03-25T18:35:40.216 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-25T18:35:40.290 INFO:teuthology.orchestra.console.smithi042:Power off completed 2024-03-25T18:35:44.221 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:35:44.331 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off 2024-03-25T18:35:44.431 INFO:teuthology.orchestra.console.smithi120:Power off completed 2024-03-25T18:35:44.536 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7890.412396669388 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7f756f4e362447a889ef0a707da1d0de status: fail success: false 2024-03-25T18:35:44.536 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:35:44.613 INFO:teuthology.run:FAIL